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FOREWORD 

Military  and  law  enforcement  personnel  are  finding  it  increasingly  necessary  to  monitor  and  control  a 
greater  number  of  devices  while  performing  their  duties.  The  need  for  methods  to  quickly  and  accurately 
shift  attention  and  control  between  individual  devices  and  their  interface  components  is  critical  to 
responding  to  complex,  asymmetric  threat  situations.  Traditionally,  manual  interactions  were  used  to 
execute  these  switching  actions,  which  can  take  seconds.  As  many  threats  can  occur  quickly,  leaving 
operators  with  only  seconds  to  take  action,  the  time  associated  with  switching  control  between  devices  to 
respond  to  such  threats  may  be  the  difference  between  success  and  failure. 

In  previous  studies  conducted  by  Popola,  Squire,  and  Liu  (2011),  researchers  sought  to  minimize  the 
time  and  workload  associated  with  switching  between  multiple  devices  by  using  eye -tracking  technology 
rather  than  manual  controls  to  perform  the  switching  action.  They  predicted  that  such  a  method  of 
interaction  would  result  in  faster  response  times,  higher  response  accuracy,  and  lower  operator  subjective 
workload.  The  research  team  successfully  integrated  a  desk-mounted  eye  tracker  plus  four  pan-tilt-zoom 
networked  cameras  into  a  simulated  surveillance  system,  called  Quick-Eye.  However,  the  results  of  the 
study  were  inconclusive;  eye-tracking  control  performed  no  better  or  worse  than  a  traditional  manual 
switching  method. 

Given  the  results  of  the  previous  study  and  lessons  learned  during  experimentation,  it  was  determined 
that  minor  improvements  to  Quick-Eye  were  needed  to  reach  system  performance  levels  predicted  for  the 
two  control  types,  and  that  minor  changes  to  the  experimental  design  and  data  collection  methods  would 
be  needed  to  improve  the  accuracy  and  precision  of  the  captured  performance  data. 

This  report  covers  the  potential  root  causes  for  the  unanticipated  performance  of  the  Quick-Eye 
system  during  previous  research  and  identifies  improvements  made  to  the  system  and  experimental  data 
collection  method.  This  report  also  highlights  the  human  computer  interaction  predictions  and  results 
used  to  test  and  validate  the  performance  of  the  Quick-Eye  system,  and  the  follow-on  results  of 
subsequent  experimental  trials. 

Based  on  results  gathered  during  the  test  and  validation  effort,  it  was  determined  the  Quick-Eye 
system  performed  as  well  or  better  than  the  predicted  performance  levels  for  both  the  eye -tracking  and 
manual  based  control  methods,  validating  the  success  of  the  integration  effort.  Furthermore,  the  results  of 
the  experimental  conditions  confirmed  that  eye-tracking  control  outperformed  manual  control  with 
respect  to  response  time,  accuracy,  and  operator  subjective  workload.  Given  these  findings,  it  is 
recommended  that  further  research  be  conducted  to  explore  the  integration  and  implementation  of 
eye -tracking  technology  as  a  means  of  interface  control,  and  to  identify  possible  task  and  system  use 
cases  relevant  to  military  and  law  enforcement  command  and  control  and  combat  systems. 

This  document  has  been  reviewed  by _ ,  Warfare  Systems  Department. 


Approved  by 


DONALD  L.  BURNETT,  Head 
Warfare  Systems  Department 
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GLOSSARY 


NSWCDD 

Naval  Surface  Warfare  Center,  Dahlgren  Division 

KLM-GOMS 

Keystroke-Level  Model  Gorals  Operators  Methods  Section 

NASA-TLX 

National  Aeronautics  and  Space  Administration  Taskload  Index 

Rt 

response  time 
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1.0  INTRODUCTION 

With  advancing  technology,  the  average  person  is  simultaneously  exposed  to  more  and  more 
information,  often  through  more  than  one  interface.  With  each  new  interface  introduced  to  their 
environment,  users  are  presented  with  greater  demands  for  their  attention  and  interaction, 
spending  more  time  switching  between  and  managing  these  various  interfaces. 

This  is  particularly  true  for  law  enforcement  and  military  personnel  who  are  equipped  with 
more  technology  than  ever  and  are  routinely  tasked  to  monitor  multiple  security  systems  and 
sensor  feeds,  sift  through  troves  of  digital  information,  or  control  various  remote  systems  and/or 
unmanned  vehicles.  With  the  addition  of  each  new  system,  sensor,  or  video  feed,  users  are 
required  to  manage  and  interact  with  increasing  numbers  of  screens  and  infonnation.  Often, 
interaction  with  an  individual  screen  or  component  of  a  system  requires  users  to  switch  control 
between  the  interfaces  of  interest  through  a  series  of  manual  manipulations. 

A  prime  example  of  such  an  interaction  is  that  of  a  security  operator  switching  between  and 
manipulating  the  various  camera  feeds  of  a  surveillance  system.  While  a  trained  and  experienced 
operator  spends  only  a  few  seconds  to  switch  control  between  feeds,  it  is  often  the  aggregation  of 
split-second  reactions  to  the  information  presented  through  the  system  that  make  the  difference 
between  mission  success  and  failure.  So  what  if  that  operator  could  have  nearly  instantaneous 
control  of  a  feed  of  interest  simply  by  looking  at  it?  How  much  time  would  be  saved  during 
execution  of  a  task,  how  much  faster  and  accurate  would  an  operator  be  able  to  perform  his 
duties  if  eye  movements  were  used  to  switch  control  between  these  devices  and  displays  rather 
than  manual  key  strokes? 

2.0  BACKGROUND 


2.1  Literature  Review 

Traditionally,  eye  tracking  has  been  primarily  used  in  the  research  worlds  of  ophthalmology, 
neurology,  and  psychology  as  a  tool  for  measuring  and  evaluating  gaze  location  relative  to  an 
individual’s  field  of  view.  The  gaze  information  collected  is  used  to  identify  oculomotor 
characteristics  and  abnormalities  and  their  relationship  to  cognition  and  mental  states  [1].  Eye 
tracking  has  also  seen  limited  use  as  an  interaction  device  for  individuals  living  with  disabilities, 
such  as  cerebral  palsy  or  paralysis,  allowing  them  to  communicate  with  and  through  computer 
interfaces  using  visual  inputs  to  type  and  interact  with  the  system  [2], 

There  are  two  primary  eye  movements,  fixations  and  saccades.  Fixations  can  be  described  as 
the  process  where  the  eyes  are  focusing  on  an  aspect  in  the  environment  for  the  purpose  of 
gaining  visual  information.  Saccades,  conversely,  are  the  rapid  transition  movements  between 
fixations  [3],  Hayhoe  and  Ballard  reviewed  several  research  studies  investigating  eye-movement 
patterns  for  various  real-world  tasks  [4].  The  central  result  was  that  eye  fixations  are  tightly 
linked  to  the  temporal  (time  related)  evolution  of  a  task,  with  very  few  task-irrelevant  areas  being 
fixated.  Furthermore,  eye  movements  are  incredibly  fast  compared  to  other  parts  of  the  body  and 
are  theoretically  the  fastest  physical  input  method  [5].  As  such,  using  eye  movements  may 
provide  an  intuitive  and  efficient  means  of  switching  between  relevant  screens  when  presented 
with  multiple  displays,  as  operators  typically  would  not  be  looking  at  a  screen  unless  it  was 
relevant  to  the  immediate  task  at  hand. 

As  eye-tracking  technology  and  gaze  prediction  methods  have  advanced,  eye  tracking  has 
become  an  increasingly  viable  means  of  human-computer  interaction.  Specifically,  those 
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interfaces  with  an  abundance  of  information  and/or  multiple  display  interfaces  could  benefit 
greatly  from  the  speed  and  accuracy  provided  by  eye  tracking  (see  Figure  1).  An  investigation  of 
the  accidental  deaths  of  23  Afghan  civilians  during  an  American  helicopter  attack  found  that  a 
Predator  drone  pilot  had  failed  to  communicate  crucial  information  regarding  the  makeup  of  the 
crowd  of  villagers.  The  primary  cause  cited  by  the  Army  and  Air  Force  for  this  failure  was 
information  overload.  The  operator  and  his  team  were  responsible  for  monitoring  the  drone’s 
multiple  video  feeds  as  well  as  communications  with  intelligence  analysts  and  troops  on  the 
ground  [6].  These  types  of  critical  work  environments  present  daunting  cognitive  and  physical 
challenges  that  leave  the  personnel  in  charge  of  these  systems  vulnerable  to  errors  as  a  result  of 
the  increased  workload,  errors  that  in  some  instances  can  have  deadly  consequences.  Eye 
tracking  as  a  method  of  control  may  be  a  solution  to  decrease  workload  and  increase  efficiency, 
providing  a  natural  and  easy  method  to  control  and  interact  with  these  systems  that  alleviates 
cognitive  and  physical  demands  and  decreases  errors. 


Figure  1.  Military  operator  monitoring  multiple  screens  (Shanker  and  Richtel,  2011) 


2.2  Previous  Results 

In  a  previous  study  conducted  by  Popola,  Squire,  and  Liu  [7],  researchers  investigated  eye 
tracking  as  a  means  of  switching  control  between  multiple  displays.  The  study  utilized  a  2x2 
factorial  design  examining  the  effects  of  “Control  Method”  (eye  tracking  or  manual)  and 
“Taskload”  (low  or  high)  on  user  performance  with  respect  to  response  time,  accuracy,  and 
workload.  The  goal  of  the  study  was  to  create  a  surveillance  system,  called  Quick-Eye,  by 
integrating  a  commercial  desk-mounted  eye-tracking  system  with  four  networked  cameras,  and 
to  evaluate  human  performance  with  the  resulting  system  during  a  simulated  threat-monitoring 
scenario.  The  experiment  was  designed  to  explore  the  benefits  of  using  eye  tracking  as  an  input 
method  and  to  determine  the  performance  differences  in  response  time,  accuracy,  and  perceived 
workload  between  eye  tracking  and  manual-based  control  methods  under  low  and  high  taskloads. 
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The  simulated  threat-monitoring  scenario  consisted  of  participants  acting  as  security 
operators  conducting  surveillance  of  office  spaces  using  four  video  feeds.  The  participants  were 
tasked  with  searching  for  potential  security  threats  in  the  fonn  of  targets  or  distracters  displayed 
on  computer  monitors  located  throughout  the  offices.  Actors  moved  throughout  each  office  space 
to  cue  the  security  operator  when  and  where  a  possible  threat  might  appear.  The  office  computer 
monitors  displayed  either  a  target  “D”  or  a  distracter  “G,”  when  the  actors  moved  from  one 
computer  to  the  next.  The  security  operator  used  directional  arrows  to  pan  the  video  feeds  left 
and  right  as  appropriate,  and  recorded  a  threat  event  using  D  or  G,  based  on  the  threat  displayed. 
After  completing  the  task  using  each  control  method,  users  were  asked  to  rate  their  perceived 
workload  using  the  NASA-Taskload  index  (NASA-TLX)  [8], 

Unexpectedly,  the  results  of  this  study  showed  no  main  or  interaction  effects  for  control 
method  or  taskload  with  respect  to  response  time,  accuracy,  or  overall  workload.  In  addition,  no 
significant  differences  in  the  means  between  eye  tracking  and  manual-based  control  were  found 
for  response  time,  accuracy,  or  overall  workload.  While  there  were  no  statistically  significant 
findings,  the  results  did  uncover  a  number  of  expected  trends  with  respect  to  the  performance  of 
the  eye-tracking  control  method.  As  anticipated,  while  using  eye -tracking  control,  threat 
responses  were  faster,  more  accurate,  and  less  workload  compared  to  manual  control,  as 
illustrated  in  Figure  2. 


Mean  (s.d.)  Response  Time  by  Control  Method  Mean  (s.d.)  Error  Rate  by  Control  Method  Mean  (s.d.)  NASA-TLX  Workload  by  ControlMethod 

■Eye-tracking  ■Manual  20%  ■Eye-tracking  ■Manual  ■Eye-tracking  BManual 


Response  Time  Error  Overall  Workload 


Figure  2.  Mean  Response  Time,  Error  Rate,  and  Subjective  Workload  by  Control  Method 

Given  these  trends,  the  lack  of  significant  findings  was  attributed  to  limitations  with  the 
system  and  the  experimental  design.  Quick-Eye,  being  a  prototype,  had  yet  to  be  fully  tested  and 
validated  from  a  performance  and  usability  perspective,  introducing  possible  system  error  and 
variability.  Furthermore,  the  system  was  only  capable  of  capturing  data  at  the  precision  level  of 
one  second,  introducing  potential  rounding  errors  in  the  data.  The  precision  level  is  particularly 
important  as  eye  movements  occur  in  fractions  of  a  second. 

In  addition  to  confounding  factors  associated  with  Quick-Eye,  the  use  of  live  actors  and  the 
lack  of  synchronized  threat  events  was  an  additional  source  of  variability.  Actor  movements 
were  hard  to  synchronize,  time,  and  repeat,  causing  variability  across  the  scenarios.  Threat 
events  were  displayed  using  timed  slideshows  that  were  also  difficult  to  synchronize  and  time, 
requiring  actors  to  simultaneously  coordinate  the  start  of  multiple  slideshows.  The  surveillance 
and  threat  systems  were  not  integrated,  requiring  researchers  to  synchronize  system  and  threat 
event  timestamps  after  the  data  collection,  exposing  the  data  analysis  to  human  data  entry  errors. 
Based  on  these  identified  limitations,  further  system  developments  and  experimental  refinements 
were  necessary  to  uncover  the  true  performance  differences  between  eye  tracking  and 
manual-based  control  methods. 
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3.0  HARDWARE  AND  SOFTWARE  INTEGRATION 

3.1  Overview 

The  Quick-Eye  system  was  developed  and  refined  over  a  two-year  period.  During  that  time 
several  experimental  trials  were  conducted  to  test  the  hardware  and  software  system  and  to 
detennine  user  performance  using  eye-tracking  technology  to  switch  control  between  multiple 
displays.  Initial  integration  efforts  focused  on  developing  a  proof  of  concept  system  to 
demonstrate  the  feasibility  of  integrating  multiple  camera  feeds  with  an  eye  tracking-controlled 
interface.  Based  on  initial  performance  data  collected,  it  was  detennined  that  further  system  and 
experimental  design  refinement  was  needed  in  order  to  accurately  compare  eye  tracking  and 
manual  control.  Eye  tracking  and  manual  control  inputs  were  developed  for  the  interface  to  allow 
for  performance  comparison  between  the  two  control  methods.  Following  each  design  and 
experimental  iteration,  the  Quick-Eye  system  was  refined  to  improve  the  performance  of  both 
eye -tracking  and  manual  control  methods  as  well  as  the  user  interface. 

An  LC  Technologies,  Inc.  Eyegaze  eye  tracker  and  Dell  Inc.  computer  system  were  used  as 
the  platform  for  the  Quick-Eye  system,  with  system  software  and  experiments  coded  in 
Microsoft  Visual  Studio  C++  and  AJAX.  A  desk-mounted  eye  tracker  was  selected  for  its  ease  of 
calibration  and  use.  Desk-mounted  eye  tracking  is  an  ideal  setup  for  situations  like  security 
monitoring,  providing  a  noninvasive  interface  that  allows  for  less  constricted  user  movement. 

The  Eyegaze  camera  works  by  directing  invisible  infrared  light  at  the  eye  and  locating  the  pupil 
by  detecting  light  reflected  off  the  retina.  Four  pan-tilt-zoom  Vivotek  Inc.  networked  cameras 
were  integrated  with  the  system  to  fonn  the  Quick-Eye  video  surveillance  component.  Each 
camera  feed  was  arranged  in  one  of  the  four  quadrants  of  the  display  and  assigned  a  number  1 
through  4  as  a  numerical  reference  for  manually  switching  between  the  different  camera  views. 
Figure  3  illustrates  this  arrangement. 


Figure  3.  Quick-Eye  System 
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3.2  System  Integration 

3.2. 1  Phase  I:  Proof  of  Concept 

Popola,  Squire,  and  Liu’s  [7]  efforts  focused  primarily  on  the  initial  development  and 
integration  of  the  eye -tracking  system  with  four  networked  cameras  and  on  developing  the  basic 
eye -tracking  and  manual-based  control  structures  used  in  experimentation.  A  basic  user  interface 
consisting  of  four  video  displays  was  developed  to  provide  a  simulated  surveillance  environment 
(see  Figure  4)  as  well  as  background  key-logging  and  timestamp  structures  for  capturing  user 
performance  data  while  using  the  system.  The  system  allowed  users  to  monitor  the  four 
independent  camera  feeds  through  one  interface,  controlling  the  camera  direction  using  the  left 
and  right  arrow  keys,  and  responding  to  onscreen  targets  using  the  D  and  G  keys.  Control  of  the 
desired  camera  feeds  was  based  on  the  control  structure,  eye  tracking  or  manual.  The 
experimental  design  required  live  actors  to  perform  simulated  office  tasks  while  potential 
on-screen  threats  appeared  on  various  monitors  within  the  space. 


Figure  4.  Interface  and  Manual  Controls 


Manual  mode  required  users  to  perfonn  three  keystrokes  in  response  to  an  on-screen  threat. 
The  first  keystroke  selects  the  camera  feed  of  interest  using  the  numerical  keys  located  in  the 
upper  row  of  the  keyboard  as  illustrated  in  Figure  5,  followed  by  a  subsequent  keystroke  to 
confirm  the  choice.  The  final  keystroke  perfonned  indicated  the  type  of  target  observed,  either  D 
or  G.  Conversely,  eye-tracking  mode  only  required  the  user  to  look  at  the  location  of  a  target  on 
the  screen  to  gain  control  of  the  appropriate  quadrant,  and  hit  the  identifying  key  relevant  to  the 
target  type.  In  each  mode,  the  system  monitored  users’  actions  by  recording  the  camera  selected, 
the  target  response  key  selected,  and  the  time  of  each  action.  This  data  was  then  output  to  a  text 
file  for  later  evaluation.  The  key  log  provided  the  selection  information  and  subsequent 
timestamp  only  when  a  user  had  confirmed  the  camera  selection  using  the  confirmation  key 
(checkmark),  and  following  the  use  of  a  target  response  key.  The  timestamps  were  recorded  to 
the  precision  of  one  second. 
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Figure  5.  Keyboard  Layout 


Following  the  initial  experimental  trials,  it  was  found  that  key  logging  and  timestamp 
structures  did  not  provide  the  level  of  fidelity  and  precision  needed  to  capture  exact  perfonnance 
of  the  two  control  methods,  as  some  user  actions  were  not  captured  or  occurred  in  fractions  of 
second.  From  a  usability  standpoint,  many  users  felt  that  there  wasn’t  sufficient  feedback  during 
the  eye-tracking  mode  and,  as  a  result,  rated  the  workload  higher.  It  was  assumed  that  a  lack  of 
experience  with  eye  tracking  and  of  an  on-screen  indicator  representing  the  location  of  their  gaze 
caused  some  users  to  not  trust  the  system  was  working  and  delaying  their  actions. 

3.2.2  Phase  II:  Refinement  and  Validation 

Based  on  the  previous  experimental  results  and  lessons  learned,  several  improvements  were 
identified  to  further  refine  and  improve  the  system’s  software  performance,  key  logging  and 
timestamp  structures,  experimental  design,  and  interface  feedback.  In  order  to  improve  the 
system’s  computing  speed  and  capabilities,  individual  software  threads  were  developed  to  handle 
the  system’s  individual  functions.  This  allowed  for  faster,  more  reliable  computing  with  less  lag 
at  the  interface  level.  Introduction  of  the  thread  structure  also  allowed  for  easier  integration  of 
improved  key  logging  and  timestamps  as  well  as  the  introduction  of  new  features  such  as  error 
logging  and  interface  feedback. 

Modifications  were  made  to  integrate  the  threat  monitoring  scenario  with  the  system  itself  to 
improve  the  overall  control  and  accuracy  of  the  experimental  design.  The  use  of  live  actors  and 
nonintegrated  threats  was  eliminated  to  decrease  experimental  variability,  improve  the  quality  of 
data  captured,  and  decrease  experimental  errors  introduced  from  synchronization  of  user  and 
threat  event  timestamp  data.  This  was  accomplished  by  creating  an  experimentation  task  thread 
that  overlaid  simulated  threats  on  the  interface,  displaying  targets  in  each  of  the  camera 
quadrants  for  users  to  locate  and  identify  (see  Figure  6). 
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Figure  6.  Experiment  Overlay 

Integrating  the  experiment  into  the  system  provided  the  ability  for  researchers  to  monitor  the 
timing  and  sequence  of  events  occurring  on  screen  as  well  as  the  user  actions  in  response  to 
those  events.  The  key  logging  and  timestamp  structures  were  redesigned  to  capture  improved 
detail  and  to  generate  output  formatted  and  ready  for  direct  analysis  following  experimentation. 
Key  logging  was  expanded  to  capture  the  type,  timing,  and  location  of  all  system  events  and  user 
actions,  and  the  precision  of  the  timestamps  was  increased  to  the  level  of  1  ms.  Table  1  shows 
that  by  recording  the  timing  and  location  of  the  threat  events,  it  was  now  possible  to  automate 
user  error  tracking,  recording  when  a  user  switched  to  the  wrong  quadrant  or  selected  the  wrong 
threat  response.  The  data  output  contained  the  threat  event  number,  system/user  action  (Eg  = 
Event  G,  Ug  =  User  response  G),  a  timestamp  (hour,  min,  second,  millisecond),  the  quadrant 
under  control  at  the  time  of  the  event/action,  the  quadrant  location  of  the  threat,  quadrant  and 
value  errors  (0  or  1),  and  the  time  between  events/actions.  It  was  possible  to  validate  the  system 
performance  and  more  accurately  compare  the  two  control  methods  by  integrating  the 
experiment  into  the  actual  system  and  improving  the  precision  and  accuracy  of  the  data 
collected. 


Table  1.  Event/Action  Timestamp  and  Error  Data  Log 


Event 

Action 

Hour 

Min 

Sec 

Ms 

Current 

Quad 

Destination 

Quad 

Quad 

Error 

Value 

Error 

Time 

Diff 

2 

Eg 

10 

43 

37 

140 

1 

2 

0 

0 

- 

2 

Switch 

10 

43 

38 

781 

2 

2 

0 

0 

1.641 

2 

Ug 

10 

43 

38 

984 

2 

2 

0 

0 

0.203 

Following  the  improvements  to  the  software  structure,  data  logging  system,  and  experimental 
design,  efforts  were  made  to  improve  the  saliency  and  feedback  of  the  eye-tracking  control 
method.  This  was  accomplished  by  incorporating  a  visual  feedback  icon  representing  the 
system’s  interpretation  of  the  location  of  the  user’s  gaze,  as  shown  in  Figure  7.  The  icon 
increased  the  saliency  for  both  the  users  and  researchers,  allowing  them  to  visually  observe  that 
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the  system  was  working  as  expected  and  improved  user  feedback  by  providing  a  real-time  cue 
that  the  system  had  recognized  users’  shift  in  gaze  to  the  quadrant  they  wished  to  control. 


Figure  7.  Eye-Tracking  Control  Visual  Feedback  Icon — Green  Dot 

After  refining  the  system,  a  pilot  study  was  conducted  to  detennine  the  impact  these  changes 
had  on  system  performance  for  each  control  method.  The  data  collected  from  this  pilot  study  was 
used  to  validate  Quick-Eye’s  performance  against  predictive  models  of  human  perfonnance  and 
to  conduct  an  initial  investigation  into  the  performance  difference  for  each  control  method. 

Based  on  these  findings,  it  was  determined  that  the  software  and  experimental  design 
refinements  were  successful,  and  that  user  performance  with  the  system  while  using  both 
eye -tracking  and  manual-based  controls  was  consistent  with  initial  predictions.  Based  on  results 
and  user  feedback  gathered  during  the  pilot  study,  additional  usability  changes  were  identified  to 
further  improve  perfonnance  of  the  system  and  the  two  control  methods. 

3.2.3  Phase  III:  Usability  Improvement 

The  usability  changes  identified  in  the  pilot  study  highlighted  disparities  in  the  control 
structures  and  feedback  between  eye  tracking  and  manual  control  methods.  While  the  data 
reflected  that  both  control  methods  performed  as  expected  based  on  the  predictive  models,  the 
comparison  of  perfonnance  with  each  control  type  showed  discrepancies  in  the  task  and 
feedback  structures  that  may  have  put  manual  control  at  a  disadvantage. 

Under  manual  control,  the  inclusion  of  a  confirmation  key  added  an  additional  step  not 
required  by  eye-tracking  control,  adding  subsequent  time  to  complete  a  switching  action.  In 
order  to  make  the  comparison  between  eye  tracking  and  manual  control  more  even,  the 
confirmation  key  was  removed,  which  allowed  users  to  switch  to  the  display  of  interest  with  a 
single  keystroke.  Based  on  users’  observations  and  feedback,  many  users  spent  time  looking 
down  at  the  keyboard  to  locate  and  select  the  correct  quadrant  key.  As  Figure  8  illustrates,  the 
quadrant  selection  key  layout  was  remapped  from  a  linear  to  a  spatial  layout  to  improve  manual 
control  usability.  Spatially  mapping  the  keys  to  the  displayed  quadrants  could  eliminate  the  need 
to  visually  search  for  the  correct  key;  therefore,  decreasing  the  switch  time,  workload,  and  errors 
associated  with  the  linear  layout. 
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Figure  8.  Manual  Control  Spatially  Mapped  Key  Layout 


In  addition  to  control  structure  discrepancies,  it  was  noted  that  the  feedback  structures  for  eye 
tracking  and  manual-based  control  were  not  equivalent.  Manual  control  had  no  visual  feedback, 
only  the  actual  action  of  hitting  the  quadrant  and  confirmation  keys  provided  users  any  indication 
of  the  quadrant  under  control.  Many  users  also  found  the  green  dot  somewhat  distracting  during 
eye -tracking  control. 

The  green  dot  was  abandoned  and  replaced  with  a  shared  visual  feedback  in  an  attempt  to 
improve  the  saliency  of  both  control  methods  and  improve  the  equivalency  of  the  comparison 
between  them.  A  highlighted  green  border  outlining  the  quadrant  was  added  as  feedback.  This 
indicated  the  current  quadrant  under  control  for  both  control  methods  (see  Figure  9).  By 
providing  the  highlight,  users  were  subtly  informed  the  system  was  responding  as  intended, 
eliminating  any  possible  condition  differences  that  could  impact  perfonnance  results.  The  border 
also  decreased  user  distraction  during  eye -tracking  control  by  shifting  the  feedback  out  of  users’ 
focal  vision  to  their  periphery  vision. 


Figure  9.  Improved  User  Feedback — Green  Border 
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4.0  METHODOLOGY 

4.1  Overview 

The  overall  objective  of  this  study  was  to  investigate  the  feasibility  and  effectiveness  of  using 
eye  tracking  as  an  input  method  for  gaining  control  of  multiple  displays  during  a  surveillance 
monitoring  scenario.  In  order  to  establish  the  feasibility  and  effectiveness  it  was  necessary,  using 
eye -tracking  and  manual-based  control  methods,  to  detennine  the: 

a.  Baseline  perfonnance  of  eye  tracking  and  manual-based  control  methods;  validate  system 
perfonnance  against  predictive  models  of  human-computer  interaction  perfonnance 

b.  Relationship  between  response  time  and  control  type  under  varying  task  conditions 

c.  Relationship  between  accuracy  and  control  type  under  varying  task  conditions 

d.  Relationship  of  user-perceived  workload  under  varying  task  conditions 

A  two-phased  approach  was  used,  consisting  of  a  validation  phase,  which  assessed  system 
perfonnance  and  piloted  the  experimental  design,  and  an  experimental  phase,  where 
experimental  trials  were  conducted  to  detennine  the  impact  of  control  method  on  user 
perfonnance. 

4.2  Participants 

Participants  were  recruited  through  a  distribution  e-mail  to  targeted  internal  Naval  Surface 
Warfare  Center,  Dahlgren  Division  (NSWCDD),  departments  and  personal  interaction  (see 
Appendix  A).  Participants  received  no  direct  or  indirect  benefit  for  participating  in  this  study.  No 
coercion  was  used  to  solicit  participation.  Prior  to  beginning  any  experiment,  participants  were 
given  an  informed  consent  fonn  and  walked  through  the  experimental  procedure  (see  Appendix 
B).  Following  their  consent,  participants  were  administered  a  brief  demographics  questionnaire 
(see  Appendix  C)  to  capture  basic  gender,  age,  and  vision  (corrected  vs.  uncorrected) 
information. 

4.3  Test  Setup 

The  study  was  conducted  in  the  NSWCDD  Human  Performance  Laboratory.  The  experiment 
setup  included  a  desktop  computer  with  an  integrated  desk-mounted  eye  tracker.  The  setup  also 
included  four  networked  pan-tilt  zoom  cameras,  providing  a  simulated  surveillance  feed. 
Participants  used  a  standard  keyboard  to  control  camera  direction,  and  used  either  the  eye  tracker 
or  keyboard  for  controlling  each  camera  feed  based  upon  the  experimental  condition.  The  eye 
tracker  was  calibrated  prior  to  use  during  the  trials. 

4.4  Phase  I:  System  Validation 

A  validation  study  was  conducted  prior  to  experimental  trials,  verifying  that  the  experimental 
design  was  capable  of  capturing  the  data  necessary  for  comparing  the  perfonnance  levels  of  the 
two  control  methods,  and  to  ensure  system  performance  matched  that  of  predicted  models  of 
performance.  To  validate  the  system  performance,  user  data  was  collected  and  evaluated  against 
the  Keystroke-Level  Model  Goals  Operators  Methods  Selection  (KLM-GOMS). 

The  KLM-GOMS  is  one  of  numerous  tools  used  in  the  design  and  evaluation  of 
human-computer  interaction  with  a  system  [8].  Specifically,  the  KLM-GOMS  is  a  simplified 
version  of  the  more  complex  GOMS  model,  using  pre-established  operators  to  predict  the  time  to 
execute  each  component  of  a  given  task. 

With  respect  to  the  Quick-Eye  system,  there  are  three  relevant  action  operators  used  in 
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switching  control  between  multiple  displays:  mental  preparation  time,  eye  movement,  and 
keystrokes.  The  KLM-GOMS  model  prediction  for  pressing  a  button  on  a  keyboard,  a  keystroke 
operator  (K),  is  0.28  seconds  for  the  average  non-secretarial  typist,  and  1.2  seconds  for  a  mental 
preparation  operator  (M)  [9].  Eye  movements,  in  comparison,  are  the  fastest  with  an  eye 
movement  operator  (E)  prediction  of  0.03  seconds  [10].  Using  these  predicted  times,  estimates  of 
event  response  time  during  the  visual  search  task  were  determined  for  eye  tracking  and  manual- 
based  control,  with  predictions  of  1.51  seconds  and  2.07  seconds  respectively  as  indicated  in 
Table  2. 


Table  2.  KLM-GOMS  Predictions  for  Manual  vs.  Eye  Movement  Camera  Switches 


Eye  Tracking  Camera  Switching 

Manual  Camera  Switching 

Mental  Preparation  [M] 

1.20  s 

Mental  Preparation  [M] 

1.20  s 

Eye  Movement  to  camera  view  to  be  engaged  [E] 

0.03  s 

Eye  Movement  to  camera  view  to  be  engaged  [E] 

0.03  s 

Keystroke  of  presented  letter  [K] 

0.28  s 

Keystroke  of  button  associated  with  desired  camera  [K] 

0.28  s 

Keystroke  of  Confirmation  Button  [K] 

0.28  s 

Keystroke  of  presented  letter  [K] 

0.28  s 

Total: 

1.51  s 

Total: 

2.07  s 

4.4.1  Task  Design 

A  single  factor,  Control  Type  (eye  tracking,  manual),  repeated  measures  experimental  design 
was  used  in  the  validation  of  the  Quick-Eye  system,  with  participants  completing  an  event- 
driven,  scripted  visual  search  task  consisting  of  7 1  events,  using  both  eye  tracking  and  manual 
input  methods.  Data  collected  from  the  scripted  task  was  analyzed  and  compared  against  KLM- 
GOMS  performance  predictions. 

The  visual  search  task  was  coded  in  Microsoft  Visual  Studio  C++,  presenting  G  and  D  event 
icons  overlaid  onto  the  Quick-Eye  surveillance  interface.  This  is  illustrated  in  Figure  10.  The 
task  consisted  of  presenting  users  with  a  single  event,  a  red  icon,  representing  a  target  or 
distracter  in  the  center  of  one  of  the  four  quadrants.  Individual  timestamps,  measured  to  the 
precision  of  1ms,  were  captured  for  each  event  occurrence  and  subsequent  actions  users  took  to 
respond  to  events  throughout  the  task.  Each  timestamp  also  included  an  error  log  that  recorded 
incorrect  user  quadrant  switches  or  target/distracter  responses.  The  timestamps  and  error  log 
were  then  used  to  automatically  calculate  event  and  user  response  speed  and  accuracy. 


Manual  Mode  Eye-Tracking  Mode 


Figure  10.  Event  Occurrence  in  Manual  and  Eye-Tracking  Modes 


Under  manual  control,  the  keyboard  was  the  sole  input  method  for  gaining  control  of 
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individual  camera  feeds  and  responding  to  the  appropriate  target  or  distracter  (see  Figure  10, 
Manual  Mode).  Users  identified  the  quadrant  based  on  the  numeric  label  and  pressed  the 
corresponding  numerical  key  at  the  top  of  the  keyboard,  followed  by  a  confirmation  key.  Users 
then  hit  the  response  key  associated  with  the  presented  icon,  with  the  next  event  occurring 
immediately  after  a  correct  response. 

Using  the  eye  tracking-based  control  method,  users  simply  looked  at  or  near  the  icon 
presented  within  a  quadrant,  and  hit  the  corresponding  response  key  of  the  target  or  distracter. 
While  using  the  eye  tracking-based  control,  users  were  presented  with  a  small  green  feedback 
dot,  increasing  saliency  of  their  gaze  location  and  reassuring  them  the  system  was  active  and 
responding  correctly  (Figure  10,  Eye-Tracking  Mode).  To  prevent  accidental  quadrant  switches 
because  of  inadvertent  eye  movements,  a  50  pixel  invisible  buffer  was  established  inside  the 
border  of  each  quadrant.  This  buffer  acted  as  an  invisible  threshold,  requiring  the  user  to  look  at 
or  near  the  icon  to  gain  control  of  the  camera  quadrant,  preventing  inadvertent  switches  when 
looking  at  or  near  the  borders. 

4.5  Phase  II:  Experiment 

Phase  II  of  the  study  conducted  experiments  using  the  validated  Quick-Eye  system  to 
determine  the  impact  of  control  type  on  user  speed,  accuracy  and  perceived  workload  under 
varying  task  conditions.  A  2  (Control  Method)  x  2  (Event  Timing)  x  2  (Taskload)  within  subjects 
repeated  measures  experimental  design  was  used  evaluating: 

a.  Experimental  task  performance  variables 

(1)  Average  event  response  time  (s) 

(2)  Average  quadrant  switch  time  (s) 

(3)  Average  target  response  time  (s) 

(4)  Quadrant  error  rate  (%) 

(5)  Value  error  rate  (%) 

b.  Subjective  workload  ratings 

(1)  Overall  subjective  workload 

(2)  Mental  and  physical  workload 

(3)  Temporal  demand 

(4)  Perfonnance  level 

(5)  Effort  and  frustration  level 

Similarly  to  the  validation  study,  users  were  asked  to  complete  a  series  of  visual  search  tasks 
using  both  the  eye  tracking  and  manual  controls.  Users  were  again  presented  with  targets  and 
distracters  and  asked  to  quickly  and  accurately  respond  to  threat  events.  Unlike  the  validation 
study  additional  factors,  Event  Timing  and  Taskload  were  manipulated  to  test  each  control 
method  under  varying  conditions  similar  to  those  found  in  other  real-world  tasks.  For  each 
control  condition,  users  were  presented  with  one  of  two  levels  for  Event  Timing,  either  event 
driven  or  randomly  timed  threat  occurrences,  and  one  of  two  levels  of  Taskload,  low  or  high. 

Event  Timing  manipulated  how  and  when  individual  threat  events  occur.  Event-driven  timing 
consisted  of  events  triggered  by  the  completion  of  the  previous  event,  with  a  new  threat  event 
appearing  the  moment  a  user  responded  to  the  previous  event  (time  between  events  =  0  sec). 
Randomly  timed  threat  occurrences  required  a  vigilance  component  with  randomized  time 
intervals  between  the  occurrences  of  each  event.  Users  were  required  to  wait  an  unknown  and 
varying  amount  of  time  after  responding  to  the  previous  event  before  the  appearance  of  the  next 
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event  (time  between  event  =  0-4  sec). 

Taskload  was  varied  for  each  combination  of  Control  Method  and  Event  Timing,  with  users 

completing  50  events  for  low  taskload  conditions  and  100  events  for  high  taskload  conditions. 

Taskload  was  intended  to  represent  varying  levels  of  workload  and  task  duration,  with  high 

taskload  placing  greater  demands  on  users  than  low  taskload. 

4.6  Statement  of  Hypothesis 

4.6.1  Main  Effects 

1 .  Control  Method  will  have  a  significant  effect  on  event  response  time,  with  faster 
response  times  using  eye  gaze-based  control 

2.  Control  Method  will  have  a  significant  effect  on  event  quadrant  selection  and  target 
response  accuracy,  with  more  accurate  quadrant  selection  and  target  response  using  eye 
gaze-based  control 

3.  Control  Method  will  have  a  significant  effect  on  overall  perceived  workload,  with 
lower  overall  perceived  workload  using  eye -based  control 

4.  Event  Timing  will  have  a  significant  effect  on  event  response  time,  with  faster  response 
times  under  event-driven  timing  conditions 

5.  Event  Timing  will  have  a  significant  effect  on  event  quadrant  selection  and  target 
response  accuracy,  with  more  accurate  quadrant  selection  and  target  response  using  eye 
gaze-based  control 

6.  Event  Timing  will  have  a  significant  effect  on  overall  perceived  workload,  with  lower 
overall  perceived  workload  under  event-driven  timing  conditions 

7.  Taskload  will  have  a  significant  effect  on  event  response  time,  with  faster  response 
times  under  low  task-loading  conditions 

8.  Taskload  will  have  a  significant  effect  on  event  quadrant  selection  and  target  response 
accuracy,  with  more  accurate  quadrant  selection  and  target  response  using  eye 
gaze-based  control 

9.  Taskload  will  have  a  significant  effect  on  overall  perceived  workload,  with  lower 
overall  perceived  workload  under  low  task-loading  conditions 

4. 6.2  Second  Order  Interaction  Effects 

10.  Control  Method  x  Event  Timing  will  have  no  significant  effect  on  event  response  time, 
quadrant  or  target  response  accuracy,  or  overall  workload 

1 1 .  Control  Method  x  Taskload  will  have  no  significant  effect  on  event  response  time, 
quadrant  or  target  response  accuracy,  or  overall  workload 

12.  Event  Timing  x  Taskload  will  have  no  significant  effect  on  event  response  time, 
quadrant  or  target  response  accuracy,  or  overall  workload 

4. 6.3  Third  Order  Interaction  Effects 

13.  Control  Method x  Event  Timing  x  Taskload  will  have  no  significant  effect  on  event 
response  time,  quadrant  or  target  response  accuracy,  or  overall  workload 
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5.0  RESULTS 

5.1  System  Validation 

5.1.1  Participation 

Twelve  participants,  five  males  and  seven  females,  ranging  in  age  from  21  to  50  (average  age 
34)  years,  with  nonnal  (20/20)  or  corrected-to-normal  vision,  participated  in  the  study.  Prior  to 
collecting  user  data,  each  participant  was  provided  with  informed  consent  and  description  of  the 
experimental  task  and  each  control  type.  Participants  were  informed  the  purpose  of  the  task  was 
to  quickly  and  accurately  switch  to  the  quadrant  with  the  displayed  threat  and  press  the 
corresponding  response  key,  with  a  subsequent  threat  appearing  following  a  correct  response. 

5.1.2  Keystroke-Level  Model  Goals  Operators  Methods  Selection 

Validation  of  the  Quick-Eye  system  was  conducted  by  comparing  the  user  performance  data 
collected  using  the  Quick-Eye  system  against  human  computer  interaction  KLM-GOMS 
perfonnance  predictions  (see  Figure  1 1).  For  the  integration  to  be  considered  a  success,  the  mean 
user  event  response  times  for  each  control  method  needed  to  be  less  than  or  equal  to  the 
predicted  times,  1.51  sec  for  eye -tracking  control  and  2.07  sec  for  manual  control.  Participant 
event  response  times  were  averaged  for  each  of  the  72  events  then  aggregated  to  determine  the 
overall  mean  event  response  time  (Rt)  for  each  control  method. 


The  overall  mean  event  Rt  for  each  control  method  was  then  compared  to  the  KLM 
prediction  using  a  one  sample  t-test  at  alpha  of  0.05.  The  mean  event  response  time  for  eye 
tracking-based  control  (M=  0.87  s,  SD  =  0. 12  s)  was  found  to  be  significantly  less  than  the 
predicted  1.51  sec,  t  (70)  =  -45 . 14,  y?  <  0.001.  The  mean  event  response  time  for  manual  control 
(M=  2.1 1  s ,  SD  =  0.27  s)  was  found  not  to  be  significantly  different  than  the  predicted  2.07  sec, 
t  (70)  =  1.20 ,p  =  0.1 15.  These  results  confirm  that  each  control  method  performed  at  or  better 
than  their  KLM-GOMS  prediction,  validating  the  Quick-Eye  system  capable  of  the  desired  levels 
of  user  performance. 

5.1.3  Pilot  Experimental  E valuation 

In  addition  to  validating  Quick-Eye’s  performance,  an  initial  exploration  into  the 
performance  of  each  control  method  was  conducted  to  evaluate  the  experimental  design  and  gain 
initial  insights  with  respect  to  the  following  hypotheses: 
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Hypothesis  1 :  Control  type  will  have  a  significant  effect  on  response  time,  with  the  eye 

tracking-based  control  having  significantly  faster  response  times  than  manual  control. 

Hypothesis  2:  Control  type  will  have  a  significant  effect  on  perceived  workload,  with  the  eye 
tracking-based  control  having  significantly  lower  perceived  workload  than  the 
manual  control. 

Hypothesis  3:  Control  type  will  have  a  significant  effect  on  accuracy,  with  the  eye 

tracking-based  control  having  significantly  fewer  errors  than  the  manual  control. 

5.1.4  Event  Response  Time 

An  initial  comparison  of  the  eye  tracking-based  and  manual  control  methods  was  perfonned 
using  a  paired  t-test  comparing  the  mean  event  response  times  of  all  12  participants.  Test  results 
found  the  methods  to  be  significantly  different,  t(l\)  =  -\\.\4,p  <  0.000,  with  the  eye 
tracking-based  control  method  (M=  0.87  s,  SD  =  0. 12  s)  being  faster  than  the  manual  control 
method  (M=  2.1 1  s,  SD  =  0.27  s),  confirming  Hypothesis  1. 

A  more  detailed  examination  of  individual  user  actions  during  an  event  response  revealed  the 
time  savings  associated  with  eye  tracking-based  control  occurred  during  the  switch  from  one 
quadrant  to  a  quadrant  of  interest  (see  Figure  12).  Comparing  the  mean  time  to  perform  a  switch 
action  using  eye  tracking-based  control  (M=  0.23  s,  SD  =  0.06  s)  to  that  of  manual  control  (M  = 
1.66  s,  SD  =  0.32  s  )  resulted  in  a  significant  difference,  t  ( 1 1)  =  -19.12  ,p<  0.001,  with  eye 
tracking-based  control  switching  taking  approximate  1.5  sec  less  time  to  perfonn. 
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Figure  12.  Mean  Time  to  Perform  Action 

Interestingly,  the  time  to  complete  a  target  response  was  significantly  different  between 
eye -tracking  and  manual  control,  t  (1 1)  =  3.46,  p  =0.003  with  manual  control  target  response 
(M  =  0.40  s,  SD  =  0.20  s)  being  slightly  faster  than  eye-tracking  control  target  response 
(M  =  0.55  s,  SD  =  0.13  s).  The  cause  of  this  difference  is  not  clear;  however,  a  possible 
explanation  could  depend  on  when  target  response  mental  preparation  occurs,  and  how  much 
mental  preparation  time  is  needed  during  the  target  response  action  for  each  control  method. 

5.1.5  Subjective  Workload 

An  examination  of  the  NASA-TLX  subjective  workload  ratings  showed  slight  differences  in 
the  perceived  workload  for  each  control  method  (see  Figure  13).  Using  a  paired  t-test,  mental 
demand  t  (\  1)  =  -3.84,  p  =  0.001,  physical  demand  t  (11)  =  -\.ll,p  =  0.052,  effort  t  (1 1)  =  -2.19, 
p  =  0.025,  and  frustration  level  t  ( 11)  =  -4.08,/?  <  0.001  were  found  to  be  significantly  different 
and  lower  for  the  eye-gaze  control  method  as  compared  to  manual  control.  A  comparison  of 


Mean  (s.d.)  Time  to  Perform  Action 

■  Bye' Tracker  ■  Manual 
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temporal  demand  t  (11)  =  0.31,;?  =  0.380,  and  performance,  t  (11)  =  -1.51,/?  =  0.08  for  each 
control  method  resulted  in  no  significant  differences.  While  not  conclusive,  eye-tracking  control 
appears  to  have  a  lower  perceived  workload  compared  to  manual  control,  consistent  with 
Hypothesis  2. 


Mean  (s.d.)  INASA-TLX  Workload  Scores 

■  EvcTrackcr  ■  Manual 


7 
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Figure  13.  Mean  Workload  Scores 


5.1. 6  Error  Rate 

Two  error  types  were  recorded:  quadrant  errors,  which  occurred  when  a  participant  switched 
to  an  incorrect  quadrant;  and  value  errors,  which  occurred  when  a  participant  selected  an 
incorrect  response  to  a  target  or  distracter.  Eye  tracking-based  control  resulted  in  a  mean 
quadrant  error  rate  of  3.52  percent  ( SD  =  5.01%)  and  mean  value  error  rate  of  2.82  percent 
(SD  =  2.88%),  with  manual  control  resulting  in  a  quadrant  error  rate  of  3. 17  percent 
(SD  =  4.58%)  and  value  error  rate  of  2.35  percent  (SD  =  2.77%),  as  shown  in  Figure  14. 


Using  paired  t-test,  no  significant  differences  were  found  in  quadrant  error  rates  t  (1 1)  =  - 
0.27,/?  =  0.79  or  value  error  rates  t  (1 1)  =  -0.46,/?  =  0.65  between  eye-tracking  and 
manual-based  control.  These  results  do  not  support  the  third  hypothesis,  and  suggest  that 
quadrant  and  value  error  rates  are  not  lower  for  eye-tracking  control.  While  this  result  is 
unexpected,  it  does  indicate  there  is  no  significant  decrement  in  performance  for  the  given  task, 
which  may  not  hold  true  for  more  complex  or  cognitively  demanding  tasks.  Further  study  is 
needed  to  determine  if  increased  cognitive  complexity  and/or  task  demand  will  further  impact 
performance  with  either  method  of  control. 
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5.2  Experimental  Evaluation 

5.2.1  Participation 

Eleven  participants,  six  males  and  five  females,  ranging  from  ages  24  to  48  (average  age  31) 
years,  with  normal  (20/20)  or  corrected-to-nonnal  vision,  participated  in  the  study.  Prior  to 
collecting  user  data,  each  participant  was  provided  with  infonned  consent  and  a  description  of 
the  experimental  task  and  each  control  type.  Participants  were  informed  the  purpose  of  the  task 
was  to  quickly  and  accurately  switch  to  the  quadrant  with  the  displayed  target  and  press  the 
corresponding  response  key,  with  a  subsequent  target  appearing  following  a  correct  response. 

5.2.2  Analysis  of  Variance  (ANO  VA ) 

A  2  (Control  Method)  x  2  (Event  Timing)  x  2  (Taskload)  analysis  of  variance  (ANOVA)  was 
calculated  for  each  of  the  following  dependent  measures: 

a.  Experimental  task  performance  variables 

(1)  Average  event  response  time  (s) 

(2)  Average  quadrant  switch  time  (s) 

(3)  Average  target  response  time  (s) 

(4)  Quadrant  error  rate  (%) 

(5)  Value  error  rate  (%) 

b.  Subjective  workload  ratings 

(1)  Overall  subjective  workload 

(2)  Mental  and  physical  workload 

(3)  Temporal  demand 

(4)  Perfonnance  level 

(5)  Effort  and  frustration  level 

5.2.3  Event  Response  Time 

An  ANOVA,  assuming  alpha  of  0.05,  was  used  to  examine  the  effects  of  Control  Method, 
Event  Timing,  and  Taskload  on  event  response  time  (see  Table  15  in  Appendix  E).  Significant 
main  effects  exist  for  two  factors,  Control  Method  F  (1,  10)  =  82.68,  p  <  0.001  and  Event  Timing 
F(  1,  10)=  12.56,  p  =  0.005),  as  well  as  their  second  order  interaction  A  (1,  10)  =  6.828,/?  = 
0.026,  as  indicated  in  Table  3.  No  significant  main  effect  exists  for  Taskload  relative  to  event 
response  time,  which  does  not  support  Hypothesis  7.  In  addition,  no  second  or  third  interaction 
effects  were  found  for  Taskload  with  respect  to  event  response  time  supporting  hypotheses  1 1 
through  13  with  respect  to  event  response  time. 


Table  3  ANOVA — Event  Response  Time  Significant  Effects 


Source 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Control  Method 

11.110 

1 

11.110 

82.680 

0.000 

Event  Timing 

0.200 

1 

0.200 

12.560 

0.005 

Control  Method  *  Event  Timing 

0.168 

1 

0.168 

6.828 

0.026 
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The  significant  main  effects,  as  illustrated  in  Figurel5,  can  be  seen  by  plotting  the  mean 
event  response  time  by  both  Control  Method  and  Event  Timing.  The  slope  for  Control  Method 
plot  shows  a  greater  impact  to  mean  response  time  compared  to  the  impact  from  Event  Timing. 


Mean  Event  Response  Time  by  Control  Method  Mean  Event  Response  Time  by  Event  Timing 


Figure  15.  Plot  of  Significant  Main  Effects  for  Mean  Response  Time  by  Control  Method  and  Event  Timing 


More  detailed  comparison  of  each  control  method  shows  a  significant  difference  in  mean 
response  time,  t  (10)  =  -9.08,/?  <  0.001  with  eye  tracking-based  control  (M=  0.88  s,  SD  =  0.04  s) 
responding  on  average  0.71  sec  faster  than  manual-based  control  (M=  1.59  s,  SD  =  0.11  s), 
supporting  Hypothesis  1  (refer  to  Figure  15).  Plotting  event  response  time  over  the  course  of  the 
simulated  surveillance  task  shows  eye  tracking  allows  for  faster,  more  consistent  response  times, 
as  shown  in  Figure  16. 
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Figure  16.  Mean  Event  Response  Time  per  Event  by  Control  Method 

Exploring  the  main  effect  plot  for  Event  Timing  shows  a  significant  difference  in  mean  event 
response  time,  t( 10)  =  3.53,/?  =  0.002  with  randomly  occurring  threat  event  (M=  1.184  s,  SE  = 
0.075  s)  response  times  being  faster  than  event  driven  occurrences  (M=  1.28  s,  SE  =  0.074  s) 
(refer  to  Figure  15).  While  the  existence  of  a  significant  main  effect  for  Event  Timing  supports 
Hypothesis  4,  the  outcome  of  that  effect  did  not  support  the  prediction  that  event  driven  threat 
occurrences  would  result  in  faster  response  times.  To  better  understand  this  outcome,  a  more 
detailed  analysis  looked  at  the  second  order  interaction  effects. 
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The  existence  of  a  second  order  interaction  effect  between  Control  Method  and  Event  type 
does  not  support  Hypothesis  10  with  respect  to  event  response  time.  The  plot  of  the  interaction 
shows  a  noticeable  negative  slope  for  manual  control  and  a  slope  of  approximately  zero  for 
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eye -tracking  control  (see  Figure  17).  During  manual-based  control,  response  times  were 
significantly  slower  for  event-driven  threat  occurrences  (M=  1.68  s,  SD  =  0.36  s)  compared  to 
randomly  timed  threat  occurrences  (M=  1.50  s,  SD  =  0.39  s),  t  (10)  =  3.43,  p  =  0.003,  and  not 
significantly  different  during  eye  tracking-based  control.  This  demonstrates  that  the  interaction 
effect  between  Control  Method  and  Event  Timing  extends  only  to  the  manual-based  control 
condition.  In  addition,  this  shows  that  the  significant  main  effect  for  Event  Timing  is  not 
independent  of  control  method  and  is  driven  by  the  interaction  effect  with  the  manual-based 
control  method. 


Menn  Event  Response  Time  by  Control  Method  and  Event  Type 

••••#•••  Eye  Control  •  Manual  Control 

2.00  -I 

1.50  \ 

I  1.00  - 

H 

0.50  - 

0.00  I 

Event  Random 

Figure  17.  Plot  of  Significant  Second  Order  Interaction  Effect  for  Mean  Response  Time 

(Control  Method  x  Event  Timing) 

5.2.4  Quadrant  Switch  and  Target  Response  Time 

Following  the  analysis  of  the  mean  event  response  time,  performance  times  for  switching 
between  quadrants  and  responding  to  the  targets  were  broken  out  and  analyzed  for  each  of  the 
main  effects,  Control  Method  and  Event  Timing  (see  Tables  16  and  17  in  Appendix  E).  It  was 
assumed  the  majority  of  time  savings  resulted  from  time  saved  during  the  switch  from  quadrant 
to  quadrant,  with  target  response  times  being  equivalent,  and  that  switching  and  target  response 
times  would  be  faster  during  randomly  driven  event  conditions. 

In  Figure  18,  a  plot  of  the  mean  switch  and  target  response  times  by  Control  Method  shows 
that  the  time  savings  associated  with  eye  tracking  based  control  occurred  during  the  switch  from 
one  quadrant  to  the  next.  Comparing  the  mean  time  to  perform  a  switch  action  using  eye 
tracking-based  control  (M  =  0.21  s,  SD  =  0.03  s)  to  that  of  manual  control  (M=  1.05  s,  SD  = 
0.24  s  )  resulted  in  a  significant  difference,  t  (10)  =  -1 1.85  ,P<  0  .001,  with  eye  tracking-based 
control  switching  occurring  approximate  0.84  sec  faster.  No  significant  difference  was  found  for 
target  response  times,  as  expected. 
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Mean  (s.d.)  Time  to  Perform  Action  by  Control 
■  Eye-Trp^kiugd  ■  Manual 


Switch  Target  Response 

Action 


Figure  18.  Mean  Time  to  Perform  Action  by  Control  Method 

A  plot  of  the  mean  switch  and  target  response  times  by  Event  Timing  in  Figure  19  again 
shows  that  user  actions  during  randomly  timed  threat  occurrence  conditions  were  faster. 
Comparing  the  mean  time  to  perform  a  switch  action  during  the  event-driven  condition  (M  = 
0.65  s,  SD  =  0.14  s)  to  that  of  random  driven  (M=  0.61  s,  SD  =  0.13  s)  resulted  in  a  significant 
difference,  t  (10)  =  3.22,  p  =  0.004,  with  switching  occurring  approximately  0.04  sec  faster. 
Comparing  the  mean  time  to  perform  a  target  response  action  during  the  event-driven  condition 
(M=  0.60  s,  SD  =  0.13  s)  to  that  of  random  driven  (M=  0.55  s,  SD  =  0.13  s)  resulted  in  a 
significant  difference,  t  (10)  =  3.65 ,p  =  0.002,  with  target  responses  occurring  approximately 
0.05  sec  faster. 


Mean  (s.d.)  Time  to  Perform  Action  by  Event  Type 

■  Event  Driven  a  Random  Driven 
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Figure  19.  Mean  Time  to  Perform  Action  by  Event  Timing 

A  plot  of  the  mean  switch  and  target  response  times  by  Control  Type  and  Event  Timing  in 
Figure  20  again  shows  that  user  actions  during  random  driven  event  conditions  were  fastest  for 
eye  tracking.  Comparing  the  mean  time  to  perform  a  switch  action  using  eye  tracking  during  the 
event-driven  condition  (M=  0.22  s,  SD  =  0.03  s)  to  that  of  random  driven  (M=  0.20  s,  SD  =  0.03 
s)  resulted  in  a  significant  difference,  t  (TO)  =  2.32, p  =  0.02.  Comparing  the  mean  time  to 
perform  a  switch  action  using  manual  control  during  the  event-driven  threat  events  (M=  1.08  s, 
SD  =  0.26  s)  to  that  of  randomly  occurring  threat  events  (M=  1.02  s,  SD  =  0.25  s)  resulted  in  a 
significant  difference,  t  (10)  =  2.75,  p  =  0.01.  Comparing  the  mean  time  to  perfonn  a  target 
response  action  using  manual  control  during  the  event-driven  condition  (M=  0.61  s,  SD  =  0.18  s) 
to  that  of  random  driven  (M=  0.51  s,  SD  =  0.1 9  s)  resulted  in  a  significant  difference,  t  (10)  = 
2.83,  p  =  0.009.  No  significant  difference  was  found  for  target  response  during  different  event 
conditions  when  using  eye  tracking. 


0.65  0.61  0.60  o  55 


Switch  Target  Response 
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Mean  (s.d.)  Time  to  Perform  Action 
for  Control  Type  by  Event  Type 


■  Event  Driven  e  Random  Driven 

2.00 


Eye-Tracking  Manual  Eye-Tracking  Manual 
Switch  Switch  Target  Response  Target  Response 


Figure  20.  Mean  Time  to  Perform  Action  by  Control  Type  by  Event  Type 
5.2.5  Error  Rates 

No  significant  main  effects  or  interaction  effects  were  found  for  Control  Method,  Event 
Timing,  and  Taskload  for  quadrant  errors  or  target  response  errors  (see  Tables  18  and  19  in 
Appendix  E).  This  result  does  not  support  our  hypotheses  that  Control  Method,  Event  Timing, 
and  Taskload  (hypotheses  2,  5,  8)  would  have  a  significant  effect  on  quadrant  and  target 
accuracy.  This  does,  however,  support  our  hypotheses  that  the  interaction  between  Control 
Method,  Event  Timing,  and  Taskload  (hypotheses  10-13)  with  respect  to  accuracy  would  have 
no  effect.  While  there  were  no  significant  effects  found,  additional  analysis  was  conducted  to 
determine  if  any  performance  trends  existed  across  condition  levels. 

A  plot  of  the  mean  percent  error  by  Control  Method  shows  that  users  switched  more 
accurately  when  using  eye  tracking-based  control  (see  Figure  21).  Comparing  the  mean  percent 
error  for  quadrant  selection  when  using  eye  tracking-based  control  (M=  1.41%,  SD  =  1.24%)  to 
that  of  manual  control  (M=  2.32%,  SD  =  1.31%)  resulted  in  a  nearly  significant  difference,  t  (10) 
=  -1.62,/?  =  0.068.  Comparing  the  mean  percent  error  for  target  response  when  using  eye 
tracking-based  control  (M=  4.59%,  SD  =  3.29%)  to  that  of  manual-based  control  (M=  4.30%, 
SD  =  2.41%)  resulted  in  no  significant  difference,  t  (10)  =  0.29,/?  =  0.39. 
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Figure  21.  Mean  Quadrant  and  Target  Response  Error  Rates  by  Control  Method 
In  Figure  22,  a  plot  of  the  mean  percent  error  by  Event  Timing  shows  that  users  were  slightly 
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more  accurate  in  switching  and  responding  to  event-driven  threat  occurrences.  Comparing  the 
mean  percent  error  for  quadrant  selection  during  event-driven  threats  (M=  1.82%,  SD  =  0.99  %) 
to  that  of  random  driven  (M=  1.91%,  SD  =  1.51%)  resulted  in  no  significant  difference,  t  (10)  = 
-0.16,  p  =  0.44.  Comparing  the  mean  percent  error  for  target  response  when  using  eye 
tracking-based  control  (M=  3.98%,  SD  =  2.42%)  to  that  of  manual  control  (M=  4.91%,  SD  = 
3.03%)  resulted  in  no  significant  difference,  t  (10)  =  -1.10, p  =  0.15. 


Mean  (s.d.)  Error  Rate  by  Event  Type 


Figure  22.  Mean  Quadrant  and  Target  Response  Error  Rates  by  Event  Type 

A  plot  of  the  mean  percent  error  by  Taskload  in  Figure  23  shows  a  trend  that  users  were  more 
accurate  in  switching  and  responding  to  threats  in  low  taskload  conditions.  Comparing  the  mean 
percent  error  for  quadrant  selection  for  low  Taskload  (M=  1.55%,  SD  =  1.08  %)  to  that  of  high 
Taskload  (M=  2.18%,  SD  =  1.30%)  resulted  in  no  significant  difference,  t  (10)  =  -1.28,/?  =  0.1 1. 
Comparing  the  mean  percent  error  for  target  response  when  using  eye  tracking-based  control 
(M=  4.36%,  SD  =  2.1 1%)  to  that  of  manual  control  (M=  4.52%,  SD  =  2.85%)  resulted  in  no 
significant  difference,  t  (10)  =  -0.31  ,p  =  0.38. 


Mean  (s.d.)  Error  Rate  by  Taskload 

■  Low  ■  High 


Figure  23.  Mean  Quadrant  and  Target  Response  Error  Rates  by  Taskload 
5.2.6  Subjective  Workload 

Examining  the  effects  of  Control  Method,  Event  Timing,  and  Taskload  on  NASA-TLX 
subjective  workload  ratings  found  several  main  and  interaction  effects  (see  Tables  20-26  in 
Appendix  E).  The  results  of  the  ANOVA  indicate  significant  main  effects  exist  for  Control 
Method  F(l,  10)  =  5.146,/>  =  0.002  and  Taskload  F  (1,  10)  =  0.455,  p  <  0.001)  with  respect  to 
overall  workload,  supporting  hypotheses  3  and  9.  No  significant  main  effect  was  found  for  Event 
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Timing  with  respect  to  overall  workload,  which  does  not  support  Hypothesis  6.  However,  a  main 
effect  for  Event  Timing  was  found  for  temporal  demand.  In  addition,  no  significant  interaction 
effects  were  found  with  respect  to  overall  workload,  supporting  hypotheses  10-13,  but 
significant  interaction  effects  were  found  between  Control  Method  and  Taskload  with  respect  to 
mental  workload,  and  Control  Method  and  Event  Timing  with  respect  to  physical  workload,  (see 
Table  4). 


Table  4.  ANOVA — Subjective  Workload  Significant  Effects 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

Overall 

5.146 

1 

5.146 

16.67 

0.002 

Mental 

9.557 

1 

9.5570 

11.49 

0.007 

Physical 

18.182 

1 

18.1820 

15.39 

0.003 

Effort 

4.545 

1 

4.5450 

6.31 

0.031 

Frustration 

2.909 

1 

2.9090 

4.10 

0.070 

Event  Timing 

Temporal 

14.727 

1 

14.7270 

14.34 

0.004 

Taskload 

Overall 

0.455 

1 

0.455 

27.76 

0.000 

Mental 

4.102 

1 

4.1020 

9.60 

0.011 

Physical 

1.136 

1 

1.1360 

6.10 

0.033 

Temporal 

4.545 

1 

4.5450 

13.16 

0.005 

Effort 

2.909 

1 

2.9090 

12.43 

0.005 

Frustration 

2.909 

1 

2.9090 

7.11 

0.024 

Control  Method  *  Taskload 

Mental 

2.557 

1 

2.5570 

19.40 

0.001 

Control  Method  *  Event  Type 

Physical 

1.636 

1 

1.6360 

6.92 

0.025 

Plotting  the  mean  response  ratings  for  overall  workload  and  each  of  the  sub  scales  by  Control 
Method  shows  a  consistent  trend  of  users  reporting  lower  perceived  workload  and  higher 
performance  when  using  eye  tracking-based  control  (see  Figure  24). 


Mean  (s.d.)  NASA- TLX  Workload  by  Control  Method 


■  Eye  T racker  ■  Manual 


Overall  Mental  Physical  Temporal  Performance  Effort  Frustration 

Workload  Demand  Demand  Demand 


Workload  Sub-Scale 


Figure  24.  Mean  Subjective  Workload  Scores  by  Control  Method 

A  similar  trend  can  be  seen  for  the  plot  of  mean  workload  response  by  Taskload  condition  in 
Figure  25,  with  users  reporting  lower  perceived  workload  under  low  taskload  conditions. 
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Mean  (s.d.)  NASA-TLX  Workload  by  Taskloading 


■  Low  Taskload  ■  High  Taskload 
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Workload  Demand  Demand  Demand 
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Figure  25.  Mean  Subjective  Workload  Scores  by  Taskload 

For  each  of  the  Control  Method  main  effects,  a  comparison  using  a  paired  t-test  at  alpha  of 
0.05  was  conducted  to  identify  significant  differences  in  mean  response  for  each  of  the 
subjective  workload  ratings.  The  comparison  in  Table  5  shows  statistically  significant 
differences  between  the  mean  subjective  workloads  for  eye  tracking  and  manual -based  control, 
with  eye  tracking  reporting  lower  perceived  workload  for  each  rating  scale. 

Table  5.  Pair-wise  Comparison  of  Mean  Subjective  Workload  Response  for  Control  Method 


Source 

Eye  Tracking 
Mean  S.D. 

Manual 
Mean  S.D 

A 

df 

t 

Sig. 

Overall 

2.11 

0.60 

2.40 

0.57 

-0.29 

10 

-3.43 

0.003 

Mental 

2.77 

1.33 

3.43 

1.37 

-0.66 

10 

-3.39 

0.003 

Physical 

1.89 

0.69 

2.80 

1.27 

-0.91 

10 

-3.92 

0.001 

Effort 

3.20 

1.48 

3.66 

1.53 

-0.45 

10 

-2.51 

0.015 

Frustration 

2.00 

0.68 

2.36 

0.88 

-0.36 

10 

-2.02 

0.035 

The  comparison  was  repeated  for  the  Taskload  main  effects  (see  Table  6).  The  comparison 
shows  a  statistically  significant  difference  between  the  mean  subjective  workloads  for  low  and 
high  taskload  conditions,  with  low  taskload  conditions  having  lower  perceived  workload  for  each 
rating  scale. 


Table  6.  Pair-wise  Comparison  of  Mean  Subjective  Workload  Response  for  Taskload 


Source 

Eye  Tracking 
Mean  S.D. 

Manual 
Mean  S.D 

A 

df 

t 

Sig. 

Overall 

2.12 

0.58 

2.39 

0.57 

-0.27 

10 

-5.40 

0.000 

Mental 

2.89 

1.36 

3.32 

1.30 

-0.43 

10 

-3.10 

0.006 

Physical 

2.23 

0.89 

2.45 

1.02 

-0.23 

10 

-2.50 

0.017 

T  emporal 

2.98 

1.02 

3.39 

1.16 

-0.41 

10 

-3.33 

0.004 

Effort 

3.25 

1.53 

3.61 

1.45 

-0.36 

10 

-3.53 

0.003 

Frustration 

2.00 

0.72 

2.36 

0.80 

-0.36 

10 

-2.67 

0.012 

Plotting  temporal  workload  for  event-driven  and  randomly  timed  threat  occurrences  shows 
the  main  effect  of  Event  Timing  (see  Figure  26).  A  direct  comparison  of  the  mean  temporal 
workload  for  event-driven  (M=  3.61,  SD  =  1.29)  and  randomly  timed  threat  task  conditions 
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(M=  2.75,  SD  =  0.93)  resulted  in  a  significant  difference,  t  (10)  =  4.14,/?  =  0.001,  with  users 
perceiving  randomly  timed  threat  occurrences  as  having  lower  temporal  demand.  This  result  is 
consistent  with  user  comments  expressing  a  feeling  of  being  rushed  during  event-driven  task 
conditions. 


Figure  26.  Mean  Subjective  Workload  Score  for  Temporal  Demand  by  Event  Timing 

Plotting  mental  workload  by  Control  Method  and  Taskload  in  Figure  27  shows  that  the 
second  order  effect  between  the  two  factors  is  driven  by  the  interaction  between  the  manual 
control  condition  and  taskload  conditions.  The  slope  of  the  line  for  manual-based  control 
indicates  an  increasing  trend  in  the  perceived  mental  demand  between  the  low  (M=  3.05,  SD  = 
1.46)  and  high  (M=  3.82,  SD  =  1.40)  taskload  conditions  with  a  significant  difference  in  the 
mean  ratings,  t  (10)  =  -5.49,/?  <  0.001.  Comparatively,  the  slope  between  the  low  (M=  2.73,  SD 
=  1.45)  and  high  (M=  2.82,  SD  =  1.44)  taskload  conditions  for  eye  tracking-based  control  is 
relatively  flat,  t(10)  =  -0.51,/?  =  0.31  with  no  significant  difference  in  the  mean  ratings. 


Figure  27.  Mean  Mental  Workload  by  Control  Method  and  Taskload 


Plotting  physical  workload  by  Control  Method  and  Event  Timing  in  Figure  28  shows 
diverging  trends  in  the  second  order  effect  between  the  two  factors.  The  slope  of  the  line  for 
manual  based  control  indicates  a  decreasing  trend  in  the  perceived  physical  workload  between 
the  event-driven  (M=  2.95,  SD  =  1.46)  and  randomly  timed  threat  task  conditions  (M=  2.64,  SD 
=  1.33).  No  significant  difference  exists  between  the  mean  ratings,  t  (10)  =  1.25,/?  =  0.12  for 
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manual  control.  Conversely,  the  slope  for  eye-tracking  control  shows  an  increasing  trend  in 
perceived  physical  workload  between  the  event-driven  (M=  1.77,  SD  =  0.75)  and  randomly 
timed  threat  conditions  (M=  2.00,  SD  =  0.87).  Unlike  manual  control,  a  significant  difference 
does  exist  between  the  mean  physical  workload  ratings  for  eye  tracking-based  control  during 
event  and  randomly  timed  threat  conditions,  t  (10)  =  -2.19,/?  =  0.03. 


Figure  28.  Mean  Physical  Workload  by  Control  Method  and  Event  Timing 

6.0  CONCLUSION 


Overall,  eye  tracking-based  control  was  found  to  be  a  faster,  less  demanding  method  for 
switching  between  multiple  displays  compared  to  manual-based  control  for  the  given  set  of 
experimental  conditions.  Eye-tracking  control,  on  average,  was  significantly  faster  by  0.71  sec 
per  event,  was  equivalently  accurate,  and  resulted  in  significantly  lower  subjective  workload 
compared  to  manual-based  control. 

As  expected,  the  time  saved  when  using  eye  tracking-based  control  was  a  direct  result  of 
decreased  manual  inputs  and  increased  speed  with  which  users  could  visually  switch  between 
displays.  In  addition  to  the  time  saved,  eye  tracking-based  control  resulted  in  more  consistent 
perfonnance  during  each  experimental  condition;  manual  control  having  much  greater  variability 
with  respect  to  event  response  time,  particularly  between  event  and  randomly  timed  threat 
conditions.  While  the  difference  of  less  than  a  second  may  not  be  particularly  impactful  for  a 
single  event,  extending  the  results  of  this  study  to  tasks  requiring  users  to  switch  hundreds  or 
thousands  of  times  between  multiple  displays  shows  an  aggregate  effect  resulting  in  substantial 
time  savings  of  minutes  to  hours. 

Furthennore,  the  results  of  this  study  show  that  users  perceived  eye  tracking-based  control  as 
less  demanding,  rating  the  associated  overall  workload  for  eye -tracking  control  lower  than 
manual  control.  This  result  is  consistent  with  the  expectation  that  a  control  method  using  a  user’s 
gaze  would  be  an  easy  and  intuitive  method  for  switching  control  between  multiple  displays, 
since  a  user  would  naturally  be  looking  at  the  display  he  or  she  intends  to  interact  with  or 
manipulate.  Removing  additional  manual  keystrokes  or  movements  decreases  physical  demands 
as  well  as  the  physical  complexity  of  the  task  that,  in  turn,  decreases  mental  demands  associated 
with  controlling  movement.  A  user  no  longer  has  to  think  of  the  correct  manual  responses  to 
switch  to  a  desired  display  but  is  able  to  simply  look  at  the  display  automatically.  This  allows 
users  to  focus  their  cognitive  processes  to  more  important  tasks  related  to  processing  information 
and  decision  making. 
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While  a  significant  difference  with  respect  to  overall  accuracy  was  not  found,  it  is  important 
to  note  that  a  nearly  significant  difference  in  quadrant  selection  accuracy  was  found  between 
eye -tracking  and  manual-based  control.  As  expected,  users  more  accurately  switched  between 
quadrants  when  using  eye  tracking-based  control  compared  to  manual  control,  as  users  simply 
needed  to  look  at  the  display  of  interest  to  correctly  switch.  During  eye-tracking  control,  errors 
occurred  only  when  a  user  looked  away  from  the  display  or  directly  at  the  border  between 
quadrants;  conversely,  during  manual  control,  users  routinely  made  keystroke  errors  switching  to 
incorrect  quadrants,  requiring  them  to  repeat  and  correct  manual  inputs.  Anecdotally,  while  users 
recovered  relatively  quickly,  recovering  from  manual  mistakes  was  much  slower  than  recovering 
from  eye-tracking  mistakes.  The  speed  advantage  provided  by  eye  tracking-based  control  not 
only  improves  the  speed  of  initial  responses  but  for  quicker  recovery  from  errors  compared  to 
manual  control. 

Ultimately,  this  study  shows  that  eye  tracking-based  control  was  a  fast,  accurate,  and  easy 
method  for  switching  between  multiple  displays.  As  the  amount  and  varieties  of  technology  in 
command  and  control  and  combat  system  environments  continues  to  increase,  users  will  be  faced 
with  greater  workload  and  demands  when  switching  between  greater  numbers  of  displays.  The 
results  of  this  study  show  that  incorporating  eye  tracking-based  control  can  alleviate  these 
demands,  providing  a  fast,  effective,  and  intuitive  alternative  to  traditional  manual  control 
methods  for  switching  between  multiple  displays. 
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Researchers  in  the  W62  Human  Systems  Integration  Branch  are  seeking  volunteers  for  a 
study  titled  “Quick-Eye:  The  effects  of  eye  gaze  based  control  on  operator  performance  in 
monitoring  multiple  displays.”  The  experiment  will  be  conducted  in  the  Human  Perfonnance 
Lab  located  in  building  1470,  from  Date  TBD  to  Date  TBD.  The  experiment  is  looking  to 
understand  the  performance  implications  of  using  a  newly  developed  interface  control  type 
utilizing  eye  tracking  to  gain  control  of  individual  monitors.  The  experiment  is  expected  to  last 
no  more  than  one  hour,  and  participation  has  no  direct  or  indirect  cost. 


Quick-Eye  System 

If  you  would  like  to  participate  or  have  any  questions  concerning  the  study  or  participation 
requirements,  please  contact:  Mr.  Patrick  Mead  at  patrick.meadl@navy.mil. 

Thank  you, 

Patrick  Mead 

Applied  Research  Scientist  and  Engineer 
Human  Systems  Integration  Branch,  W-62 
NSWCDD  Dahlgren,  VA  22448 
E-mail:  patrick.meadl@navy.mil 
Office:  540-653-5186 
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1.  Introduction: 

You  are  being  asked  to  voluntarily  participate  in  a  research  study  entitled  " Examination  of  Human  Performance 
Characteristics  Using  Eye  Tracking  and  Manual  Based  Control  Systems  for  Monitoring  Multiple  Displays 
Thank  you  for  your  willingness  to  participate  in  this  experiment.  Your  participation  will  provide  important 
information  to  us  regarding  the  use  of  the  eye  tracking  technology  in  monitoring  scenarios.  By  volunteering,  you  are 
assisting  us  in  obtaining  valuable  information. 

2.  Purpose  of  the  study:  Operators  are  increasingly  responsible  for  monitoring  multiple  screen  feeds  from 
numerous  devices  and  may  be  required  to  shift  control  quickly  among  these  devices  when  confronted  with  complex, 
asymmetric  threat  situations.  This  switching  is  done  manually,  which  can  take  seconds;  however,  a  threat  event  can 
occur  quickly,  leaving  the  operator  with  only  a  brief  window  to  respond.  The  goal  is  to  evaluate  eye-tracking 
technology  and  the  use  gaze  information  as  an  alternative  to  manual  input  for  switching  between  multiple  devices. 

3.  Procedures  to  be  followed 

You  will  experience  eight  conditions  during  two  scenario  types  to  study  the  effects  of  input  method  on  task  load, 
mental  demand,  workload,  and  accuracy.  You  will  perform  the  entire  experiment  in  the  Human  Performance  Lab 
(HPL)  in  building  1470. 

4.  Discomforts  and  Risks: 

As  a  result  of  participating  in  this  study,  participants  will  be  exposed  to  no  additional  risk  beyond  the  normal  risk 
present  at  their  workplace. 

5.  Benefits:  The  benefits  to  society  and  you  are  described  below: 

(a)  Benefits  to  You:  You  will  receive  no  benefits  directly;  however,  you  will  be  participating  in  a  continued  study  to 
investigate  the  eye  tracking  technology  in  a  way  that  has  not  been  done  before. 

(b)  Potential  Benefits  to  Society:  The  sponsor  (NISE)  will  get  a  proof  of  concept,  possible  patents,  research  papers. 
This  study  could  lead  to  future  larger  scale  studies  and  can  serve  as  a  baseline  to  pave  the  way.  The  eye  tracking 
technology  used  in  this  study  could  potentially  provide  new  ways  to  decrease  operator  workload  and  increase  the 
ability  to  detect  targets  faster  and  with  more  precision. 

6.  Duration/Time  of  the  Procedures  and  Study: 

Each  participant  will  need  no  more  than  one  hour  to  complete  this  study. 

7.  Alternative  Procedures  that  could  be  utilized: 

N/A 

8.  Statement  of  Confidentiality: 

Every  reasonable  effort  will  be  made  to  keep  your  responses  and  identity  confidential.  The  U.S.  Navy  may  keep, 
preserve,  use  in  any  manner,  and  dispose  of  the  findings  of  this  evaluation,  including  your  input  or  opinions. 
Information  collected  during  this  study  is  subject  to  the  Privacy  Act  of  1974,  which  is  described  below: 

The  Privacy  Act  of  1974,  5  U.S.C.  §  552a,  establishes  a  code  of  fair  information  practices  that  governs  the 
collection,  maintenance,  use,  and  dissemination  of  personally  identifiable  information  about  individuals  that  is 
maintained  in  systems  of  records  by  federal  agencies.  A  system  of  records  is  a  group  of  records  under  the  control  of 
an  agency  from  which  information  is  retrieved  by  the  name  of  the  individual  or  by  some  identifier  assigned  to  the 
individual.  The  Privacy  Act  requires  that  agencies  give  the  public  notice  of  their  systems  of  records  by  publication  in 
the  Federal  Register.  The  Privacy  Act  prohibits  the  disclosure  of  information  from  a  system  of  records  absent  the 
written  consent  of  the  subject  individual,  unless  the  disclosure  is  pursuant  to  one  of  twelve  statutory  exceptions.  The 
Act  also  provides  individuals  with  a  means  by  which  to  seek  access  to  and  amendment  of  their  records,  and  sets 
forth  various  agency  recordkeeping  requirements. 

The  tenets  of  the  Privacy  Act,  SECNAVINST  521 1.5E,  will  be  followed.  Only  the  Department  of  the  Navy, 
Department  of  Defense,  and  other  U.S.  Government  agencies  will  use  the  information  gained  from  the  studies 
described  in  the  protocol,  provided  the  use  is  compatible  with  the  purpose  for  which  the  information  was  collected. 
Anyone  needing  to  use  this  data  for  purposes  other  than  what  is  described  on  this  informed  consent  form  will  need 
to  obtain  authorization  from  each  participant  to  use  the  data  in  the  proposed  manner.  Any  reports  or  publications 
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containing  data  resulting  from  studies  will  not  identify  you  by  name  or  initials  unless  your  express  permission  is 
obtained.  The  Commander  of  the  Naval  Surface  Warfare  Center,  Dahlgren  Division  may  grant  approval  for  use  of 
the  information  to  non-Government  agencies  or  individuals. 

The  objective  and  subjective  information  gathered  in  this  study  is  considered  Personally  Identifiable  Information 
(PII),  and  will  be  handled  and  stored  in  accordance  with  NSWCDD  Instruction  521 1. 1C  (Privacy  Act  Program). 
Hardcopy  data  will  be  stored  in  a  locked  container  in  Building  1470  for  a  period  of  three  years;  informed  consent 
and  collected  data  sheets  will  be  physically  separated.  Electronic  data  will  be  password  protected. 

By  participating  in  this  study,  you  agree  to  maintain  in  strict  confidence  all  information  disclosed  to  you,  and  you 
agree  not  to  use  directly  or  indirectly  for  your  own  benefit,  or  the  benefit  of  other  third  parties,  any  information 
presented  at  the  time  of  this  study. 


9.  Right  to  Ask  Questions 


You  have  a  right  to  ask  questions  at  any  time  before,  during,  or  after  the  test.  Please  contact  the  Principal 
Investigator,  one  of  the  Associate  Investigators,  the  medical  monitor,  or  the  Institutional  Review  Board  (1RB)  chair 
at  any  time  with  questions,  complaints,  or  concerns  about  the  research.  They  are: 

Principal  Investigator:  Patrick  Mead  IRB  Chair:  Meredith  Bondurant 

Naval  Surface  Warfare  Center  -  Naval  Surface  Warfare  Center  - 

Dahlgren  Division  Dahlgren  Division 


ATTN:  Code  W62 


ATTN:  Code  Z20 


18444  Frontage  Road,  Suite  327 
Dahlgren,  VA  22448-5161 
(540)  653-5186 
patrick.meadl@navy.mil 


4045  Higley  Road  Suite  344 
Dahlgren,  VA  22448-5162 
(540)  653-9201 

meredith.bondurant@navy.mil 


10.  Payment  for  Participation:  There  is  no  cost  to  you  for  participating  in  this  study.  Your  participation  in  this 
study  is  voluntary,  refusing  to  participate  will  involve  no  penalty,  and  you  may  discontinue  your  participation  at  any 
time  without  consequence.  The  principal  investigator  reserves  the  right  to  stop  and  end  participation  at  any  point  if  it 
is  in  the  best  interest  of  the  participant. 


11.  Cost  of  Participating:  There  is  no  cost  directly  or  indirectly  for  participation  in  this  study. 


12.  Voluntary  Participation:  Your  participation  in  this  study  is  voluntary.  You  may  withdraw  from  this  study  at 
any  time  by  notifying  any  of  the  investigators.  You  may  decline  to  answer  any  or  all  questions.  Refusal  to  take  part 
in,  or  withdrawing  from,  this  study  will  involve  no  penalty  or  loss  of  benefits  you  would  otherwise  receive. 


13.  Injury  Clause:  There  is  no  additional  risk  anticipated  to  participate  in  this  study,  however,  if  you  feel  that  you 
have  incurred  a  research  related  injury,  contact  the  principal  investigator. 

14.  Participation  Requirements:  We  are  looking  for  civilian  and  active  duty  personnel  stationed  at  the  Naval 
Surface  Warfare  Center  Dahlgren  with  normal  or  corrected  to  normal  vision.  Please  remember,  you  cannot  fail  any 
part  of  this  study,  and  there  is  no  right  or  wrong  answer.  The  purpose  of  this  study  is  to  evaluate  eye  gaze  based 
control.  We  are  not  in  any  way  evaluating  your  abilities. 


If  you  agree  to  take  part  in  this  research  study  and  the  information  outlined  above,  please  sign  your  name  and 
indicate  the  date  below.  By  signing  below,  you  are  also  certifying  that  you  have  been  informed  of  the  information 
above  and  that  your  participation  in  this  study  is  voluntary.  You  will  be  given  a  copy  of  this  signed  and  dated 
consent  form  for  your  records. 


Participant’s  Name 


Participant’s  signature 


Date 


Investigator’s  Name 


Investigator’s  signature 


Date 
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APPENDIX  C:  DEMOGRAPHICS  QUESTIONNAIRE 
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Demographics  Questionnaire  (Experimenter  Use  Only): 

Participant  Number: _ 

Glasses:  Yes _  No _ 

Age: _ 

Gender: 
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APPENDIX  D:  NASA-TLX  SUBJECTIVE  WORKLOAD  QUESTIONNAIRE 
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Condition  1 — Mode: _ 

Mental  Demand:  How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking,  deciding,  calculating, 
remembering,  looking,  searching,  etc.)?  Was  the  task  easy  or  demanding,  simple  or  complex,  exacting  or  forgiving? 

1  2  3  4  5  6  7  8 

Low . High 

Physical  Demand:  How  physical  activity  was  required  (e.g.,  pushing,  pulling,  turning,  controlling,  activating,  etc.)? 
Was  the  task  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  restful  or  laborious? 

1  2  3  4  5  6  7  8 

Low - High 

Temporal  Demand:  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the  tasks  or  task 
elements  occurred?  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

1  2  3  4  5  6  7  8 

Low - High 

Performance:  How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  task  set  by  the  experimenter 
(or  yourself)?  How  satisfied  were  you  with  your  performance  in  accomplishing  these  goals? 

1  2  3  4  5  6  7  8 

Good  - . -Poor 

Effort:  How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your  level  of  performance? 

1  2  3  4  5  6  7  8 

Low - High 

Frustration:  How  insecure,  discouraged,  irritated,  stressed  and  annoyed  versus  secure,  gratified,  content,  relaxed, 
and  complacent  did  you  feel  during  the  task? 

1  2  3  4  5  6  7  8 

Low - High 
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APPENDIX  E:  DATA  TABLES 
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Experimental  Evaluation  Raw  Data  per  Measure 

Table  7.  Aggregated  Raw  Data  per  Measure 


Avg.  Event 
Response 

Time  (s) 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

0.77 

0.89 

0.74 

0.82 

1.49 

1.94 

1.31 

1.40 

2 

0.74 

0.74 

0.74 

0.71 

1.12 

1.54 

1.22 

1.23 

3 

1.01 

1.10 

0.95 

1.08 

2.10 

2.29 

2.12 

1.90 

4 

0.89 

0.89 

1.06 

0.89 

1.23 

1.72 

1.82 

1.09 

5 

0.82 

0.89 

0.84 

0.77 

1.79 

1.47 

1.77 

1.46 

6 

0.94 

0.97 

1.05 

0.94 

1.78 

2.25 

1.68 

1.36 

7 

0.83 

1.06 

0.73 

0.85 

1.46 

1.66 

1.16 

1.35 

8 

0.77 

0.92 

1.05 

0.89 

2.22 

1.63 

1.40 

1.90 

9 

0.73 

0.63 

0.81 

0.80 

1.25 

1.45 

1.08 

1.12 

10 

0.72 

0.74 

0.72 

0.72 

1.13 

1.05 

0.91 

1.02 

11 

1.22 

1.10 

0.97 

1.07 

2.23 

2.13 

2.35 

2.26 

Avg.  Switch 
Time  (s) 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

0.20 

0.22 

0.16 

0.21 

0.88 

1.17 

0.80 

0.95 

2 

0.21 

0.21 

0.23 

0.24 

0.82 

1.11 

0.92 

0.91 

3 

0.25 

0.26 

0.19 

0.19 

1.26 

1.53 

1.25 

1.25 

4 

0.19 

0.23 

0.07 

0.20 

0.78 

1.13 

1.12 

0.67 

5 

0.21 

0.21 

0.20 

0.21 

1.34 

1.12 

1.35 

1.13 

6 

0.27 

0.25 

0.23 

0.24 

1.27 

1.42 

1.23 

1.11 

7 

0.25 

0.21 

0.18 

0.20 

0.89 

1.00 

0.82 

0.99 

8 

0.25 

0.25 

0.25 

0.23 

1.30 

1.03 

0.90 

1.23 

9 

0.09 

0.21 

0.19 

0.17 

0.72 

0.84 

0.70 

0.74 

10 

0.21 

0.21 

0.17 

0.21 

0.67 

0.61 

0.67 

0.71 

11 

0.26 

0.25 

0.24 

0.21 

1.44 

1.50 

1.52 

1.50 

Avg.  Target 
Response 

Time  (s) 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

0.51 

0.62 

0.49 

0.54 

0.63 

0.85 

0.54 

0.43 

2 

0.49 

0.49 

0.51 

0.49 

0.29 

0.40 

0.26 

0.29 

3 

0.75 

0.83 

0.69 

0.80 

0.90 

0.86 

0.95 

0.77 

4 

0.55 

0.54 

0.48 

0.55 

0.49 

0.67 

0.77 

0.46 

5 

0.53 

0.64 

0.54 

0.53 

0.30 

0.34 

0.37 

0.27 

6 

0.65 

0.70 

0.79 

0.66 

0.58 

0.90 

0.52 

0.36 

7 

0.54 

0.67 

0.53 

0.59 

0.51 

0.63 

0.41 

0.47 

8 

0.53 

0.68 

0.60 

0.61 

0.93 

0.64 

0.56 

0.78 

9 

0.42 

0.40 

0.55 

0.47 

0.51 

0.58 

0.38 

0.40 

10 

0.47 

0.48 

0.49 

0.49 

0.50 

0.42 

0.31 

0.37 

11 

0.86 

0.83 

0.74 

0.80 

0.77 

0.63 

0.80 

0.71 
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a.  Table  7  (cont’d) 


Quadrant 
Error  Rate 
(%) 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

0.00% 

0.00% 

0.00% 

0.00% 

2.00% 

2.00% 

2.00% 

10.00% 

2 

1.00% 

0.00% 

0.00% 

0.00% 

0.00% 

2.00% 

3.00% 

2.00% 

3 

0.00% 

2.00% 

0.00% 

2.00% 

2.00% 

0.00% 

3.00% 

0.00% 

4 

3.00% 

2.00% 

7.00% 

0.00% 

1.00% 

0.00% 

2.00% 

0.00% 

5 

5.00% 

2.00% 

2.00% 

2.00% 

9.00% 

0.00% 

1.00% 

0.00% 

6 

0.00% 

0.00% 

0.00% 

0.00% 

3.00% 

2.00% 

7.00% 

2.00% 

7 

0.00% 

2.00% 

0.00% 

2.00% 

5.00% 

4.00% 

1.00% 

0.00% 

8 

0.00% 

0.00% 

8.00% 

0.00% 

2.00% 

4.00% 

3.00% 

2.00% 

9 

3.00% 

0.00% 

2.00% 

8.00% 

5.00% 

2.00% 

5.00% 

6.00% 

10 

1.00% 

0.00% 

0.00% 

0.00% 

0.00% 

4.00% 

0.00% 

2.00% 

11 

6.00% 

2.00% 

0.00% 

0.00% 

2.00% 

0.00% 

0.00% 

0.00% 

Value  Error 
Rate  (%) 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

12.00% 

8.00% 

10.00% 

12.00% 

5.00% 

2.00% 

7.00% 

8.00% 

2 

6.00% 

6.00% 

4.00% 

2.00% 

2.00% 

2.00% 

4.00% 

2.00% 

3 

2.00% 

2.00% 

7.00% 

8.00% 

4.00% 

2.00% 

4.00% 

4.00% 

4 

12.00% 

6.00% 

18.00% 

8.00% 

3.00% 

4.00% 

5.00% 

10.00% 

5 

7.00% 

0.00% 

6.00% 

8.00% 

5.00% 

2.00% 

9.00% 

10.00% 

6 

0.00% 

2.00% 

2.00% 

4.00% 

1.00% 

2.00% 

0.00% 

0.00% 

7 

6.00% 

6.00% 

1.00% 

2.00% 

8.00% 

12.00% 

8.00% 

2.00% 

8 

1.00% 

0.00% 

6.00% 

4.00% 

7.00% 

2.00% 

1.00% 

2.00% 

9 

1.00% 

4.00% 

1.00% 

2.00% 

4.00% 

12.00% 

7.00% 

8.00% 

10 

3.00% 

4.00% 

2.00% 

2.00% 

4.00% 

4.00% 

0.00% 

6.00% 

11 

0.00% 

0.00% 

1.00% 

4.00% 

0.00% 

0.00% 

3.00% 

2.00% 

Overall 

Workload 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

2.71 

2.57 

2.29 

1.57 

2.57 

2.86 

2.43 

2.14 

2 

3.57 

3.57 

2.57 

2.00 

3.43 

3.00 

3.29 

2.43 

3 

3.00 

3.43 

3.00 

3.14 

3.43 

2.57 

2.86 

3.00 

4 

2.29 

2.71 

3.14 

1.86 

2.71 

2.86 

3.14 

2.29 

5 

1.71 

1.57 

1.71 

1.29 

2.00 

1.86 

1.71 

1.71 

6 

1.29 

1.43 

1.00 

1.00 

1.43 

1.00 

1.86 

1.14 

7 

2.43 

2.71 

2.00 

1.57 

2.71 

2.71 

2.14 

2.43 

8 

1.53 

1.10 

2.25 

1.71 

2.38 

1.33 

1.48 

1.62 

9 

2.43 

1.86 

2.14 

1.71 

2.86 

2.71 

3.00 

2.57 

10 

2.16 

2.00 

2.49 

1.87 

3.43 

3.29 

2.43 

2.29 

11 

1.73 

1.21 

1.60 

1.83 

2.43 

1.56 

2.22 

2.10 
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Table  7  (conUd) 


Mental 

Workload 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

2 

4 

3 

2 

4 

4 

4 

3 

2 

5 

4 

4 

3 

5 

5 

5 

3 

3 

5 

5 

5 

6 

5 

4 

5 

5 

4 

2 

4 

5 

4 

4 

4 

5 

3 

5 

1 

1 

1 

1 

2 

1 

2 

1 

6 

1 

1 

1 

1 

1 

1 

3 

1 

7 

2 

3 

2 

2 

3 

4 

3 

3 

8 

2 

1 

3 

2 

3 

2 

3 

2 

9 

3 

3 

3 

3 

5 

5 

6 

4 

10 

3 

4 

5 

3 

6 

4 

5 

5 

11 

2 

1 

2 

2 

3 

1 

2 

2 

Physical 

Workload 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

2 

1 

2 

1 

4 

3 

4 

3 

2 

3 

3 

3 

3 

5 

5 

5 

3 

3 

3 

2 

3 

4 

3 

4 

3 

5 

4 

2 

1 

2 

1 

3 

3 

1 

2 

5 

1 

1 

1 

1 

1 

1 

1 

1 

6 

1 

1 

1 

1 

1 

1 

1 

1 

7 

2 

1 

3 

1 

3 

2 

2 

3 

8 

2 

1 

2 

2 

2 

1 

2 

2 

9 

2 

2 

2 

2 

4 

4 

2 

2 

10 

3 

2 

3 

2 

4 

6 

4 

5 

11 

2 

1 

2 

2 

2 

3 

3 

3 

Temporal 

Workload 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

4 

5 

2 

1 

3 

4 

3 

3 

2 

5 

4 

4 

3 

5 

6 

5 

3 

3 

4 

5 

4 

3 

3 

6 

3 

5 

4 

3 

3 

4 

1 

6 

5 

4 

2 

5 

2 

2 

2 

2 

2 

2 

2 

1 

6 

2 

3 

1 

1 

1 

2 

2 

1 

7 

3 

5 

2 

2 

2 

3 

2 

2 

8 

3 

2 

3 

2 

2 

3 

2 

3 

9 

5 

4 

4 

2 

3 

3 

3 

4 

10 

6 

5 

5 

3 

7 

6 

4 

5 

11 

3 

2 

2 

3 

2 

3 

3 

3 
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b.  Table  7  (cont’d) 


Performance 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

4 

6 

4 

4 

4 

4 

4 

4 

2 

3 

2 

4 

5 

5 

4 

4 

4 

3 

6 

5 

5 

4 

6 

5 

5 

6 

4 

6 

6 

5 

6 

6 

6 

6 

6 

5 

6 

6 

5 

6 

6 

5 

5 

4 

6 

7 

6 

7 

6 

6 

6 

5 

6 

7 

5 

5 

5 

6 

4 

5 

6 

5 

8 

6 

6 

4 

6 

6 

5 

6 

5 

9 

5 

6 

6 

5 

4 

3 

3 

4 

10 

7 

6 

7 

6 

6 

6 

7 

6 

11 

6 

6 

6 

5 

6 

5 

5 

6 

Effort 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

4 

3 

4 

1 

5 

3 

3 

3 

2 

5 

5 

4 

3 

5 

5 

5 

4 

3 

6 

6 

6 

6 

6 

6 

5 

6 

4 

4 

6 

5 

4 

4 

4 

6 

5 

5 

1 

1 

1 

1 

1 

1 

1 

1 

6 

2 

2 

1 

1 

1 

2 

3 

1 

7 

4 

4 

4 

2 

4 

5 

4 

4 

8 

2 

2 

3 

3 

2 

3 

2 

2 

9 

4 

3 

4 

3 

5 

4 

4 

4 

10 

4 

3 

4 

3 

6 

6 

5 

4 

11 

2 

1 

2 

2 

2 

3 

3 

3 

Frustration 

Ey-Ev-Hi 

Ey-Ev-Lw 

Ey-Rd-Hi 

Ey-Rd-Lw 

Mn-Ev-Hi 

Mn-Ev-Lw 

Mn-Rd-Hi 

Mn-Rd-Lw 

1 

4 

3 

2 

2 

3 

2 

2 

2 

2 

4 

4 

2 

2 

3 

3 

3 

3 

3 

2 

1 

2 

2 

2 

2 

3 

2 

4 

1 

1 

2 

1 

2 

2 

3 

1 

5 

1 

1 

1 

1 

1 

1 

1 

1 

6 

1 

1 

1 

1 

1 

1 

1 

1 

7 

3 

3 

2 

1 

4 

3 

2 

3 

8 

2 

1 

3 

3 

1 

4 

1 

1 

9 

3 

2 

3 

2 

3 

4 

4 

3 

10 

2 

1 

3 

3 

5 

5 

3 

1 

11 

2 

2 

2 

2 

2 

3 

3 

3 
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Mean,  Standard  Deviation,  and  Confidence  Interval  Data  per  Measure 


Table  8.  Mean,  Standard  Deviation,  and  Confidence  Interval  (Control  Method) 


Measure 

Control 

Method 

95%  Confidence  Interval  | 

M 

SD 

LB 

UB 

Avg.  Event  Response 

Eye-tracking 

0.88 

0.04 

0.79 

0.96 

Time  (s) 

Manual 

1.59 

0.11 

1.34 

1.84 

Avg.  Switch  Time  (s) 

Eye-tracking 

0.21 

0.01 

0.19 

0.23 

Manual 

1.05 

0.08 

0.88 

1.22 

Avg.  Target  Response 

Eye-tracking 

0.59 

0.04 

0.52 

0.67 

Time  (s) 

Manual 

0.56 

0.05 

0.44 

0.68 

Quadrant  Error  Rate 

Eye-tracking 

1.00% 

0.00% 

1.00% 

2.00% 

Manual 

2.00% 

0.00% 

1.00% 

3.00% 

Value  Error  Rate 

Eye-tracking 

5.00% 

1.00% 

2.00% 

7.00% 

Manual 

4.00% 

1.00% 

3.00% 

6.00% 

Overall  Workload 

Eye-tracking 

2.42 

0.26 

1.85 

2.99 

Manual 

2.90 

0.30 

2.23 

3.58 

Mental  Workload 

Eye-tracking 

2.77 

0.40 

1.88 

3.67 

Manual 

3.43 

0.41 

2.52 

4.35 

Physical  Workload 

Eye-tracking 

1.89 

0.21 

1.42 

2.35 

Manual 

2.80 

0.38 

1.94 

3.65 

Temporal  Workload 

Eye-tracking 

3.05 

0.31 

2.37 

3.73 

Manual 

3.27 

0.39 

2.41 

4.13 

Performance 

Eye-tracking 

5.41 

0.26 

4.83 

5.99 

Manual 

5.11 

0.26 

4.53 

5.70 

Effort 

Eye-tracking 

3.21 

0.45 

2.21 

4.20 

Manual 

3.66 

0.46 

2.63 

4.69 

Fiustration 

Eye-tracking 

2.00 

0.21 

1.54 

2.46 

Manual 

2.36 

0.27 

1.77 

2.96 

Table  9.  Mean,  Standard  Deviation,  and  Confidence  Interval  (Event  Timing) 


Measure 

Event  Timing 

95%  Confidence  Interval  | 

M 

SD 

LB 

UB 

Avg.  Event  Response 

Event 

1.28 

0.07 

1.12 

1.45 

Time  (s) 

Random 

1.18 

0.08 

1.02 

1.35 

Avg.  Switch  Time  (s) 

Event 

0.65 

0.04 

0.56 

0.75 

Random 

0.61 

0.04 

0.52 

0.70 

Avg.  Target  Response 

Event 

0.60 

0.04 

0.51 

0.70 

Time  (s) 

Random 

0.55 

0.04 

0.46 

0.64 

Quadrant  Error  Rate 

Event 

2.00% 

0.00% 

1.00% 

3.00% 

Random 

2.00% 

1.00% 

1.00% 

3.00% 

Value  Error  Rate 

Event 

4.00% 

1.00% 

2.00% 

6.00% 

Random 

5.00% 

1.00% 

3.00% 

7.00% 

Overall  Workload 

Event 

2.76 

0.31 

2.06 

3.46 

Random 

2.56 

0.24 

2.01 

3.10 

Mental  Workload 

Event 

3.07 

0.41 

2.15 

3.99 

Random 

3.14 

0.39 

2.26 

4.01 

Physical  Workload 

Event 

2.36 

0.30 

1.69 

3.04 

Random 

2.32 

0.29 

1.67 

2.97 

Temporal  Workload 

Event 

3.57 

0.41 

2.65 

4.49 

Random 

2.75 

0.28 

2.13 

3.37 

Performance 

Event 

5.32 

0.27 

4.73 

5.91 

Random 

5.21 

0.23 

4.70 

5.71 

Effort 

Event 

3.57 

0.48 

2.51 

4.63 

Random 

3.30 

0.43 

2.34 

4.25 

Fiustration 

Event 

2.32 

0.28 

1.70 

2.94 

Random 

2.05 

0.19 

1.63 

2.46 
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Table  10.  Mean,  Standard  Deviation,  and  Confidence  Interval  (Taskload) 


Measure 

Taskload 

95%  Confidence  Interval  j 

M 

SD 

LB 

UB 

Avg.  Event  Response 

Low 

1.24 

0.07 

1.09 

1.40 

Time  (s) 

High 

1.22 

0.08 

1.05 

1.40 

Avg.  Switch  Time  (s) 

Low 

0.65 

0.04 

0.56 

0.73 

High 

0.62 

0.05 

0.52 

0.72 

Avg.  Target  Response 

Low 

0.58 

0.04 

0.49 

0.67 

Time  (s) 

High 

0.57 

0.04 

0.47 

0.67 

Quadrant  Error  Rate 

Low 

2.00% 

0.00% 

1.00% 

2.00% 

High 

2.00% 

0.00% 

1.00% 

3.00% 

Value  Error  Rate 

Low 

4.00% 

1.00% 

3.00% 

6.00% 

High 

5.00% 

1.00% 

3.00% 

6.00% 

Overall  Workload 

Low 

2.50 

0.27 

1.90 

3.10 

High 

2.82 

0.28 

2.20 

3.44 

Mental  Workload 

Low 

2.89 

0.41 

1.98 

3.80 

High 

3.32 

0.39 

2.44 

4.19 

Physical  Workload 

Low 

2.23 

0.27 

1.63 

2.83 

High 

2.46 

0.31 

1.77 

3.14 

Temporal  Workload 

Low 

2.93 

0.33 

2.20 

3.67 

High 

3.39 

0.35 

2.60 

4.17 

Performance 

Low 

5.32 

0.21 

4.86 

5.78 

High 

5.21 

0.28 

4.60 

5.82 

Effort 

Low 

3.25 

0.46 

2.22 

4.28 

High 

3.61 

0.44 

2.64 

4.59 

Fiustration 

Low 

2.00 

0.22 

1.51 

2.49 

High 

2.36 

0.24 

1.83 

2.90 
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Table  11.  Mean,  Standard  Deviation,  and  Confidence  Interval  (Control  Method  x  Event  Timing) 


Measure 

Event 

Control  Method  Timing 

95%  Confidence  Interval 

M 

SD 

LB 

UB 

Avg.  Event  Response 

Eye-tracking 

Event 

0.88 

0.04 

0.78 

0.98 

Time  (s) 

Random 

0.87 

0.04 

0.79 

0.95 

Manual 

Event 

1.68 

0.11 

1.43 

1.92 

Random 

1.50 

0.12 

1.23 

1.76 

Avg.  Switch  Time  (s) 

Eye-tracking 

Event 

0.22 

0.01 

0.20 

0.24 

Random 

0.20 

0.01 

0.18 

0.22 

Manual 

Event 

1.08 

0.08 

0.91 

1.26 

Random 

1.02 

0.08 

0.85 

1.19 

Avg.  Target  Response 

Eye-tracking 

Event 

0.60 

0.04 

0.51 

0.69 

Time  (s) 

Random 

0.59 

0.03 

0.52 

0.66 

Manual 

Event 

0.61 

0.06 

0.48 

0.73 

Random 

0.51 

0.06 

0.38 

0.64 

Quadrant  Error  Rate 

Eye-tracking 

Event 

1.00% 

0.00% 

0.00% 

2.00% 

Random 

2.00% 

1.00% 

0.00% 

3.00% 

Manual 

Event 

2.00% 

0.00% 

1.00% 

3.00% 

Random 

2.00% 

1.00% 

1.00% 

4.00% 

Value  Error  Rate 

Eye-tracking 

Event 

4.00% 

1.00% 

2.00% 

6.00% 

Random 

5.00% 

1.00% 

3.00% 

8.00% 

Manual 

Event 

4.00% 

1.00% 

2.00% 

6.00% 

Random 

5.00% 

1.00% 

3.00% 

7.00% 

Overall  Workload 

Eye-tracking 

Event 

2.49 

0.30 

1.83 

3.14 

Random 

2.35 

0.25 

1.80 

2.90 

Manual 

Event 

3.04 

0.35 

2.25 

3.83 

Random 

2.76 

0.26 

2.19 

3.34 

Mental  Workload 

Eye-tracking 

Event 

2.68 

0.41 

1.78 

3.59 

Random 

2.86 

0.43 

1.91 

3.82 

Manual 

Event 

3.46 

0.44 

2.46 

4.45 

Random 

3.41 

0.39 

2.54 

4.28 

Physical  Workload 

Eye-tracking 

Event 

1.77 

0.20 

1.34 

2.21 

Random 

2.00 

0.23 

1.48 

2.52 

Manual 

Event 

2.96 

0.43 

2.00 

3.91 

Random 

2.64 

0.38 

1.80 

3.47 

Temporal  Workload 

Eye-tracking 

Event 

3.55 

0.39 

2.68 

4.41 

Random 

2.55 

0.27 

1.94 

3.16 

Manual 

Event 

3.59 

0.50 

2.48 

4.70 

Random 

2.96 

0.31 

2.28 

3.63 

Performance 

Eye-tracking 

Event 

5.50 

0.34 

4.75 

6.25 

Random 

5.32 

0.24 

4.79 

5.84 

Manual 

Event 

5.14 

0.26 

4.55 

5.72 

Random 

5.09 

0.29 

4.46 

5.73 

Effort 

Eye-tracking 

Event 

3.36 

0.48 

2.29 

4.44 

Random 

3.05 

0.44 

2.07 

4.02 

Manual 

Event 

3.77 

0.51 

2.64 

4.91 

Random 

3.55 

0.44 

2.56 

4.54 

Frustration 

Eye-tracking 

Event 

2.05 

0.32 

1.33 

2.76 

Random 

1.96 

0.21 

1.49 

2.42 

Manual 

Event 

2.59 

0.35 

1.81 

3.37 

Random 

2.14 

0.26 

1.55 

2.72 
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Table  12.  Mean,  Standard  Deviation,  and  Confidence  Interval  (Control  Method  x  Taskload) 


Measure 

Control  Method  Taskload 

95%  Confidence  Interval 

M 

SD 

LB 

UB 

Avg.  Event  Response 

Eye-tracking 

Low 

0.89 

0.04 

0.80 

0.97 

Time  (s) 

High 

0.87 

0.04 

0.78 

0.95 

Manual 

Low 

1.60 

0.11 

1.37 

1.83 

High 

1.57 

0.12 

1.30 

1.85 

Avg.  Switch  Time  (s) 

Eye-tracking 

Low 

0.22 

0.01 

0.21 

0.23 

High 

0.21 

0.01 

0.18 

0.23 

Manual 

Low 

1.08 

0.08 

0.91 

1.24 

High 

1.03 

0.08 

0.85 

1.21 

Avg.  Target 

Eye-tracking 

Low 

0.61 

0.04 

0.53 

0.69 

Response  Time  (s) 

High 

0.58 

0.03 

0.50 

0.65 

Manual 

Low 

0.56 

0.05 

0.45 

0.66 

High 

0.56 

0.06 

0.42 

0.69 

Quadrant  Error  Rate 

Eye-tracking 

Low 

1.00% 

0.00% 

0.00% 

2.00% 

High 

2.00% 

1.00% 

0.00% 

3.00% 

Manual 

Low 

2.00% 

1.00% 

1.00% 

3.00% 

High 

3.00% 

1.00% 

2.00% 

4.00% 

Value  Error  Rate 

Eye-tracking 

Low 

4.00% 

1.00% 

3.00% 

6.00% 

High 

5.00% 

1.00% 

2.00% 

8.00% 

Manual 

Low 

5.00% 

1.00% 

3.00% 

6.00% 

High 

4.00% 

1.00% 

3.00% 

6.00% 

Overall  Workload 

Eye-tracking 

Low 

2.25 

0.25 

1.70 

2.80 

High 

2.58 

0.27 

1.98 

3.18 

Manual 

Low 

2.74 

0.31 

2.05 

3.44 

High 

3.06 

0.30 

2.39 

3.73 

Mental  Workload 

Eye-tracking 

Low 

2.73 

0.42 

1.79 

3.67 

High 

2.82 

0.40 

1.93 

3.71 

Manual 

Low 

3.05 

0.42 

2.10 

3.99 

High 

3.82 

0.41 

2.90 

4.74 

Physical  Workload 

Eye-tracking 

Low 

1.64 

0.23 

1.11 

2.16 

High 

2.14 

0.21 

1.66 

2.61 

Manual 

Low 

2.82 

0.35 

2.03 

3.60 

High 

2.77 

0.43 

1.82 

3.73 

Temporal  Workload 

Eye-tracking 

Low 

2.77 

0.29 

2.13 

3.42 

High 

3.32 

0.37 

2.51 

4.13 

Manual 

Low 

3.09 

0.43 

2.14 

4.04 

High 

3.46 

0.37 

2.63 

4.28 

Performance 

Eye-tracking 

Low 

5.41 

0.24 

4.87 

5.95 

High 

5.41 

0.32 

4.69 

6.13 

Manual 

Low 

5.23 

0.26 

4.66 

5.80 

High 

5.00 

0.30 

4.33 

5.67 

Effort 

Eye-tracking 

Low 

2.96 

0.46 

1.92 

3.99 

High 

3.46 

0.46 

2.44 

4.47 

Manual 

Low 

3.55 

0.50 

2.43 

4.66 

High 

3.77 

0.44 

2.78 

4.76 

Fiustration 

Eye-tracking 

Low 

1.82 

0.19 

1.39 

2.25 

High 

2.18 

0.23 

1.68 

2.69 

Manual 

Low 

2.18 

0.28 

1.56 

2.81 

High 

2.55 

0.30 

1.88 

3.21 
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Table  13.  Mean,  Standard  Deviation,  and  Confidence  Interval  (Event  Timing  x  Taskload) 


Measure 

Event  Timing  Taskload 

95%  Confidence  Interval 

M 

SD 

LB 

UB 

Avg.  Event  Response 

Event 

Low 

1.32 

0.08 

1.15 

1.49 

Time  (s) 

High 

1.24 

0.08 

1.06 

1.42 

Random 

Low 

1.17 

0.08 

1.00 

1.33 

High 

1.20 

0.08 

1.02 

1.39 

Avg.  Switch  Time  (s) 

Event 

Low 

0.68 

0.04 

0.58 

0.78 

High 

0.63 

0.05 

0.52 

0.73 

Random 

Low 

0.61 

0.04 

0.53 

0.70 

High 

0.61 

0.05 

0.51 

0.71 

Avg.  Target 

Event 

Low 

0.63 

0.04 

0.53 

0.72 

Response  Time  (s) 

High 

0.58 

0.05 

0.48 

0.68 

Random 

Low 

0.54 

0.04 

0.44 

0.63 

High 

0.56 

0.04 

0.46 

0.66 

Quadrant  Error  Rate 

Event 

Low 

1.00% 

0.00% 

1.00% 

2.00% 

High 

2.00% 

1.00% 

1.00% 

4.00% 

Random 

Low 

2.00% 

1.00% 

0.00% 

3.00% 

High 

2.00% 

1.00% 

1.00% 

3.00% 

Value  Error  Rate 

Event 

Low 

4.00% 

1.00% 

2.00% 

6.00% 

High 

4.00% 

1.00% 

2.00% 

6.00% 

Random 

Low 

5.00% 

1.00% 

3.00% 

7.00% 

High 

5.00% 

1.00% 

3.00% 

7.00% 

Overall  Workload 

Event 

Low 

2.63 

0.33 

1.90 

3.36 

High 

2.90 

0.31 

2.20 

3.59 

Random 

Low 

2.36 

0.25 

1.81 

2.92 

High 

2.75 

0.25 

2.18 

3.32 

Mental  Workload 

Event 

Low 

3.00 

0.46 

1.98 

4.02 

High 

3.14 

0.41 

2.23 

4.04 

Random 

Low 

2.77 

0.40 

1.88 

3.67 

High 

3.50 

0.41 

2.59 

4.41 

Physical  Workload 

Event 

Low 

2.18 

0.28 

1.56 

2.81 

High 

2.55 

0.35 

1.77 

3.32 

Random 

Low 

2.27 

0.32 

1.56 

2.98 

High 

2.36 

0.30 

1.71 

3.02 

Temporal  Workload 

Event 

Low 

3.36 

0.44 

2.38 

4.35 

High 

3.77 

0.40 

2.88 

4.67 

Random 

Low 

2.50 

0.30 

1.83 

3.17 

High 

3.00 

0.32 

2.30 

3.71 

Performance 

Event 

Low 

5.41 

0.25 

4.85 

5.97 

High 

5.23 

0.30 

4.55 

5.91 

Random 

Low 

5.23 

0.21 

4.77 

5.69 

High 

5.18 

0.26 

4.60 

5.77 

Effort 

Event 

Low 

3.50 

0.51 

2.36 

4.64 

High 

3.64 

0.45 

2.63 

4.65 

Random 

Low 

3.00 

0.44 

2.02 

3.99 

High 

3.59 

0.44 

2.62 

4.56 

Fiustration 

Event 

Low 

2.14 

0.30 

1.46 

2.81 

High 

2.50 

0.29 

1.85 

3.16 

Random 

Low 

1.86 

0.18 

1.46 

2.26 

High 

2.23 

0.23 

1.72 

2.73 
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Table  14.  Mean,  Standard  Deviation,  and  Confidence  Interval  (Control  Method  x  Event  Timing  x  Taskload) 


Measure 

Control  Method 

Event 

Timing 

Taskload 

95%  Confidence  Interval 

M 

SD 

LB 

UB 

Avg.  Event  Response 

Eye-tracking 

Event 

Low 

0.90 

0.05 

0.80 

1.01 

Time  (s) 

High 

0.86 

0.05 

0.76 

0.96 

Random 

Low 

0.87 

0.04 

0.78 

0.95 

High 

0.88 

0.04 

0.78 

0.97 

Manual 

Event 

Low 

1.74 

0.12 

1.48 

2.00 

High 

1.62 

0.13 

1.33 

1.91 

Random 

Low 

1.46 

0.12 

1.20 

1.73 

High 

1.53 

0.14 

1.22 

1.84 

Avg.  Switch  Time  (s) 

Eye-tracking 

Event 

Low 

0.23 

0.01 

0.21 

0.24 

High 

0.22 

0.02 

0.18 

0.25 

Random 

Low 

0.21 

0.01 

0.20 

0.22 

High 

0.19 

0.00 

0.16 

0.23 

Manual 

Event 

Low 

1.13 

0.08 

0.95 

1.32 

High 

1.03 

0.09 

0.84 

1.23 

Random 

Low 

1.02 

0.08 

0.84 

1.19 

High 

1.03 

0.09 

0.84 

1.22 

Avg.  Target 

Eye-tracking 

Event 

Low 

0.63 

0.04 

0.53 

0.72 

Response  Time  (s) 

High 

0.57 

0.04 

0.49 

0.66 

Random 

Low 

0.59 

0.04 

0.52 

0.67 

High 

0.58 

0.03 

0.51 

0.66 

Manual 

Event 

Low 

0.63 

0.06 

0.50 

0.76 

High 

0.58 

0.06 

0.44 

0.73 

Random 

Low 

0.48 

0.06 

0.36 

0.61 

High 

0.53 

0.07 

0.39 

0.68 

Quadrant  Error  Rate 

Eye-tracking 

Event 

Low 

0.90% 

0.30% 

0.20% 

1.60% 

High 

1.70% 

0.70% 

0.30% 

3.20% 

Random 

Low 

1.30% 

0.70% 

0.00% 

2.90% 

High 

1.70% 

0.90% 

0.00% 

3.70% 

Manual 

Event 

Low 

1.80% 

0.50% 

0.70% 

2.90% 

High 

2.80% 

0.80% 

1.00% 

4.60% 

Random 

Low 

2.20% 

1.00% 

0.10% 

4.30% 

High 

2.50% 

0.60% 

1.00% 

3.90% 

Value  Error  Rate 

Eye-tracking 

Event 

Low 

3.50% 

0.90% 

1.50% 

5.40% 

High 

4.50% 

1.30% 

1.60% 

7.50% 

Random 

Low 

5.10% 

1.00% 

2.80% 

7.40% 

High 

5.30% 

1.60% 

1.80% 

8.70% 

Manual 

Event 

Low 

4.00% 

1.20% 

1.20% 

6.80% 

High 

3.90% 

0.70% 

2.30% 

5.50% 

Random 

Low 

4.90% 

1.10% 

2.50% 

7.30% 

High 

4.40% 

1.00% 

2.20% 

6.50% 

Overall  Workload 

Eye-tracking 

Event 

Low 

2.40 

0.04 

1.72 

3.08 

High 

2.58 

0.30 

1.92 

3.24 

Random 

Low 

2.11 

0.25 

1.55 

2.66 

High 

2.59 

0.27 

1.98 

3.20 

Manual 

Event 

Low 

2.87 

0.38 

2.02 

3.71 

High 

3.21 

0.35 

2.44 

3.99 

Random 

Low 

2.62 

0.28 

2.01 

3.23 

High 

2.91 

0.27 

2.30 

3.52 
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Table  14  (cont'd) 


Measure 

Control  Method 

Event 

Timing 

Taskload 

95%  Confidence  Interval 

M 

SD 

LB 

UB 

Mental  Workload 

Eye-tracking 

Event 

Low 

2.82 

0.46 

1.79 

3.85 

High 

2.55 

0.41 

1.63 

3.47 

Random 

Low 

2.64 

0.43 

1.67 

3.60 

High 

3.09 

0.46 

2.07 

4.11 

Manual 

Event 

Low 

3.18 

0.48 

2.11 

4.26 

High 

3.73 

0.45 

2.73 

4.73 

Random 

Low 

2.91 

0.42 

1.99 

3.83 

High 

3.91 

0.42 

2.99 

4.83 

Physical  Workload 

Eye-tracking 

Event 

Low 

1.46 

0.21 

0.99 

1.92 

High 

2.09 

0.21 

1.62 

2.56 

Random 

Low 

1.82 

0.30 

1.16 

2.48 

High 

2.18 

0.23 

1.68 

2.69 

Manual 

Event 

Low 

2.91 

0.39 

2.04 

3.78 

High 

3.00 

0.51 

1.88 

4.12 

Random 

Low 

2.73 

0.41 

1.82 

3.63 

High 

2.55 

0.41 

1.63 

3.47 

Temporal  Workload 

Eye-tracking 

Event 

Low 

3.46 

0.46 

2.44 

4.47 

High 

3.64 

0.39 

2.77 

4.50 

Random 

Low 

2.09 

0.25 

1.53 

2.65 

High 

3.00 

0.38 

2.15 

3.85 

Manual 

Event 

Low 

3.27 

0.57 

2.00 

4.55 

High 

3.91 

0.48 

2.85 

4.97 

Random 

Low 

2.91 

0.42 

1.99 

3.83 

High 

3.00 

0.30 

2.33 

3.67 

Performance 

Eye-tracking 

Event 

Low 

5.46 

0.37 

4.64 

6.27 

High 

5.55 

0.37 

4.73 

6.36 

Random 

Low 

5.36 

0.24 

4.82 

5.91 

High 

5.27 

0.33 

4.53 

6.01 

Manual 

Event 

Low 

5.36 

0.28 

4.74 

5.99 

High 

4.91 

0.29 

4.28 

5.53 

Random 

Low 

5.09 

0.29 

4.46 

5.73 

High 

5.09 

0.34 

4.33 

5.85 

Effort 

Eye-tracking 

Event 

Low 

3.27 

0.54 

2.07 

4.48 

High 

3.46 

0.46 

2.44 

4.47 

Random 

Low 

2.64 

0.45 

1.63 

3.65 

High 

3.46 

0.47 

2.40 

4.51 

Manual 

Event 

Low 

3.73 

0.57 

2.45 

5.01 

High 

3.82 

0.44 

2.74 

4.89 

Random 

Low 

3.36 

0.47 

2.31 

4.42 

High 

3.73 

0.45 

2.73 

4.73 

Frustration 

Eye-tracking 

Event 

Low 

1.82 

0.33 

1.09 

2.54 

High 

2.27 

0.33 

1.53 

3.01 

Random 

Low 

1.82 

0.23 

1.31 

2.32 

High 

2.09 

0.21 

1.62 

2.56 

Manual 

Event 

Low 

2.46 

0.39 

1.59 

3.32 

High 

2.73 

0.38 

1.87 

3.58 

Random 

Low 

1.91 

0.29 

1.28 

2.54 

High 

2.36 

0.31 

1.67 

3.05 
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Analysis  of  Variance  (ANOVA)  Tables  per  Measure 

Table  15.  ANOVA:  Event  Response  Time 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

11.110 

1 

11.110 

82.680 

0.000 

Event  Timing 

0.200 

1 

0.200 

12.560 

0.005 

Taskload 

0.011 

1 

0.011 

0.463 

0.512 

Control  Method  *  Event  Timing 

0.168 

1 

0.168 

6.828 

0.026 

Control  Method  *  Taskload 

0.001 

1 

0.001 

0.048 

0.831 

Event  Timing  *  Taskload 

0.081 

1 

0.081 

1.925 

0.195 

Control  Method  *  Event  Timing  *  Taskload 

0.024 

1 

0.024 

0.487 

0.501 

Error  (Control  Method) 

1.344 

10 

0.134 

Error  (Event  Timing) 

0.159 

10 

0.016 

Error  (Taskload) 

0.231 

10 

0.023 

Error  (Control  Method  *  Event  Timing) 

0.247 

10 

0.025 

Error  (Control  Method  *  Taskload) 

0.125 

10 

0.013 

Error  (Event  Timing  *  Taskload) 

0.421 

10 

0.042 

Error  (Control  Method  *  Event  Timing  *  Taskload) 

0.491 

10 

0.049 

Table  16.  ANOVA:  Switch  Time 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

15.540 

1 

15.5400 

140.698 

0.000 

Event  Type 

0.380 

1 

0.3800 

9.986 

0.010 

Taskioad 

0.020 

1 

0.0200 

2.677 

0.133 

Control  Method  *  Event  Type 

0.009 

1 

0.0090 

2.983 

0.115 

Control  Method  *  Taskload 

0.005 

1 

0.0050 

0.675 

0.430 

Event  Type  *  Taskload 

0.014 

1 

0.0140 

1.042 

0.331 

Control  Method  *  Event  Type  *  Taskload 

0.018 

1 

0.0180 

1.089 

0.321 

Error  (Control  Method) 

1.104 

10 

0.1104 

Error  (Event  Type) 

0.039 

10 

0.0039 

Error  (Taskload) 

0.074 

10 

0.0074 

Error  (Control  Method  *  Event  Type) 

0.030 

10 

0.0030 

Error  (Control  Method  *  Taskload) 

0.078 

10 

0.0078 

Error  (Event  Type  *  Taskload) 

0.132 

10 

0.0132 

Error  (Control  Method  *  Event  Type  *  Taskload) 

0.166 

10 

0.0166 
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Table  17.  ANOVA:  Target  Response  Time 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

0.029 

1 

0.0290 

1.024 

0.335 

Event  Type 

0.065 

1 

0.0650 

13.759 

0.004 

Taskload 

0.005 

1 

0.0050 

1.549 

0.242 

Control  Method  *  Event  Type 

0.041 

1 

0.0410 

3.839 

0.079 

Control  Method  *  Taskload 

0.006 

1 

0.0060 

1.212 

0.297 

Event  Type  *  Taskload 

0.027 

1 

0.0270 

2.548 

0.141 

Control  Method  *  Event  Type  *  Taskload 

0.004 

1 

0.0040 

0.362 

0.561 

Error  (Control  Method) 

0.288 

10 

0.0288 

Error  (Event  Type) 

0.047 

10 

0.0047 

Error  (Taskload) 

0.031 

10 

0.0031 

Error  (Control  Method  *  Event  Type) 

0.108 

10 

0.0108 

Error  (Control  Method  *  Taskload) 

0.053 

10 

0.0053 

Error  (Event  Type  *  Taskload) 

0.104 

10 

0.0104 

Error  (Control  Method  *  Event  Type  *  Taskload) 

0.117 

10 

0.0117 

Table  18.  ANOVA:  Quadrant  Errors 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

0.002 

1 

0.0020 

2.614 

0.137 

Event  Type 

0.000 

1 

0.0002 

0.026 

0.875 

Taskload 

0.001 

1 

0.0010 

1.639 

0.229 

Control  Method  *  Event  Type 

0.000 

1 

0.0000 

0.047 

0.834 

Control  Method  *  Taskload 

0.000 

1 

0.0000 

0.000 

1.000 

Event  Type  *  Taskload 

0.000 

1 

0.0000 

0.230 

0.642 

Control  Method  *  Event  Type  *  Taskload 

0.000 

1 

0.0000 

0.070 

0.797 

Error  (Control  Method) 

0.007 

10 

0.0007 

Error  (Event  Type) 

0.007 

10 

0.0007 

Error  (Taskload) 

0.005 

10 

0.0005 

Error  (Control  Method  *  Event  Type) 

0.004 

10 

0.0004 

Error  (Control  Method  *  Taskload) 

0.006 

10 

0.0006 

Error  (Event  Type  *  Taskload) 

0.007 

10 

0.0007 

Error  (Control  Method  *  Event  Type  *  Taskload) 

0.003 

10 

0.0003 
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Table  19.  ANOVA:  Target  Response  Errors 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

0.000 

1 

0.0000 

0.087 

0.774 

Event  Type 

0.002 

1 

0.0020 

1.218 

0.296 

Taskload 

0.000 

1 

0.0001 

0.094 

0.766 

Control  Method  *  Event  Type 

0.000 

1 

0.0000 

0.233 

0.640 

Control  Method  *  Taskload 

0.001 

1 

0.0010 

0.733 

0.412 

Event  Type  *  Taskload 

0.000 

1 

0.0000 

0.417 

0.533 

Control  Method  *  Event  Type  *  Taskload 

0.000 

1 

0.0000 

0.064 

0.805 

Error  (Control  Method) 

0.022 

10 

0.0022 

Error  (Event  Type) 

0.016 

10 

0.0016 

Error  (Taskload) 

0.006 

10 

0.0006 

Error  (Control  Method  *  Event  Type) 

0.006 

10 

0.0006 

Error  (Control  Method  *  Taskload) 

0.007 

10 

0.0007 

Error  (Event  Type  *  Taskload) 

0.006 

10 

0.0006 

Error  (Control  Method  *  Event  Type  *  Taskload) 

0.004 

10 

0.0004 

Table  20.  ANOVA:  Overall  Workload 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

5.146 

1 

5.1460 

16.672 

0.002 

Event  Type 

0.941 

1 

0.9410 

2.413 

0.151 

Taskload 

2.317 

1 

2.3170 

29.385 

0.000 

Control  Method  *  Event  Type 

0.101 

1 

0.1010 

0.469 

0.509 

Control  Method  *  Taskload 

0.001 

1 

0.0010 

0.007 

0.934 

Event  Type  *  Taskload 

0.080 

1 

0.0800 

0.283 

0.606 

Control  Method  *  Event  Type  *  Taskload 

0.180 

1 

0.1800 

2.352 

0.156 

Error  (Control  Method) 

3.087 

10 

0.3087 

Error  (Event  Type) 

3.900 

10 

0.3900 

Error  (Taskload) 

0.789 

10 

0.0789 

Error  (Control  Method  *  Event  Type) 

2.154 

10 

0.2154 

Error  (Control  Method  *  Taskload) 

1.253 

10 

0.1253 

Error  (Event  Type  *  Taskload) 

2.839 

10 

0.2839 

Error  (Control  Method  *  Event  Type  *  Taskload) 

0.765 

10 

0.0765 
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Table  21.  ANOVA:  Mental  Workload 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

9.557 

1 

9.5570 

11.489 

0.007 

Event  Type 

0.102 

1 

0.1020 

0.177 

0.683 

Taskload 

4.102 

1 

4.1020 

9.601 

0.011 

Control  Method  *  Event  Type 

0.284 

1 

0.2840 

1.359 

0.271 

Control  Method  *  Taskload 

2.557 

1 

2.5570 

19.397 

0.001 

Event  Type  *  Taskload 

1.920 

1 

1.9200 

2.761 

0.128 

Control  Method  *  Event  Type  *  Taskload 

0.102 

1 

0.1020 

0.194 

0.669 

Error  (Control  Method) 

8.318 

10 

0.8318 

Error  (Event  Type) 

5.773 

10 

0.5773 

Error  (Taskload) 

4.273 

10 

0.4273 

Error  (Control  Method  *  Event  Type) 

2.091 

10 

0.2091 

Error  (Control  Method  *  Taskload) 

1.318 

10 

0.1318 

Error  (Event  Type  *  Taskload) 

6.955 

10 

0.6955 

Error  (Control  Method  *  Event  Type  *  Taskload) 

5.273 

10 

0.5273 

Table  22.  ANOVA:  Physical  Workload 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

18.182 

1 

18.1820 

15.385 

0.003 

Event  Type 

0.045 

1 

0.0450 

0.076 

0.788 

Taskload 

1.136 

1 

1.1360 

6.098 

0.033 

Control  Method  *  Event  Type 

1.636 

1 

1.6360 

6.923 

0.025 

Control  Method  *  Taskload 

1.636 

1 

1.6360 

3.750 

0.082 

Event  Type  *  Taskload 

0.409 

1 

0.4090 

0.732 

0.412 

Control  Method  *  Event  Type  *  Taskload 

0.000 

1 

0.0000 

0.000 

1.000 

Error  (Control  Method) 

11.818 

10 

1.1818 

Error  (Event  Type) 

5.955 

10 

0.5955 

Error  (Taskload) 

1.864 

10 

0.1864 

Error  (Control  Method  *  Event  Type) 

2.364 

10 

0.2364 

Error  (Control  Method  *  Taskload) 

4.364 

10 

0.4364 

Error  (Event  Type  *  Taskload) 

5.591 

10 

0.5591 

Error  (Control  Method  *  Event  Type  *  Taskload) 

3.000 

10 

0.3000 
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Table  23.  ANOVA:  Temporal  Demand 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

1.136 

1 

1.1360 

1.445 

0.257 

Event  Type 

14.727 

1 

14.7270 

14.336 

0.004 

Taskload 

4.545 

1 

4.5450 

13.158 

0.005 

Control  Method  *  Event  Type 

0.727 

1 

0.7270 

0.879 

0.371 

Control  Method  *  Taskload 

0.182 

1 

0.1820 

0.233 

0.640 

Event  Type  *  Taskload 

0.045 

1 

0.0450 

0.057 

0.816 

Control  Method  *  Event  Type  *  Taskload 

2.227 

1 

2.2270 

2.865 

0.121 

Error  (Control  Method) 

7.864 

10 

0.7864 

Error  (Event  Type) 

10.273 

10 

1.0273 

Error  (Taskload) 

3.455 

10 

0.3455 

Error  (Control  Method  *  Event  Type) 

8.273 

10 

0.8273 

Error  (Control  Method  *  Taskload) 

7.818 

10 

0.7818 

Error  (Event  Type  *  Taskload) 

7.955 

10 

0.7955 

Error  (Control  Method  *  Event  Type  *  Taskload) 

7.773 

10 

0.7773 

Table  24.  ANOVA:  Performance 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

1.920 

1 

1.9200 

1.753 

0.215 

Event  Type 

0.284 

1 

0.2840 

0.558 

0.472 

Taskload 

0.284 

1 

0.2840 

0.919 

0.360 

Control  Method  *  Event  Type 

0.102 

1 

0.1020 

0.194 

0.669 

Control  Method  *  Taskload 

0.284 

1 

0.2840 

0.401 

0.541 

Event  Type  *  Taskload 

0.102 

1 

0.1020 

0.577 

0.465 

Control  Method  *  Event  Type  *  Taskload 

0.557 

1 

0.5570 

1.047 

0.330 

Error  (Control  Method) 

10.955 

10 

1.0955 

Error  (Event  Type) 

5.091 

10 

0.5091 

Error  (Taskload) 

3.091 

10 

0.3091 

Error  (Control  Method  *  Event  Type) 

5.273 

10 

0.5273 

Error  (Control  Method  *  Taskload) 

7.091 

10 

0.7091 

Error  (Event  Type  *  Taskload) 

1.773 

10 

0.1773 

Error  (Control  Method  *  Event  Type  *  Taskload) 

5.318 

10 

0.5318 
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Table  25.  ANOVA:  Effort 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

4.545 

1 

4.5450 

6.309 

0.031 

Event  Type 

1.636 

1 

1.6360 

2.677 

0.133 

Taskload 

2.909 

1 

2.9090 

12.427 

0.005 

Control  Method  *  Event  Type 

0.045 

1 

0.0450 

0.073 

0.792 

Control  Method  *  Taskload 

0.409 

1 

0.4090 

0.557 

0.473 

Event  Type  *  Taskload 

1.136 

1 

1.1360 

2.463 

0.148 

Control  Method  *  Event  Type  *  Taskload 

0.182 

1 

0.1820 

0.879 

0.371 

Error  (Control  Method) 

7.205 

10 

0.7205 

Error  (Event  Type) 

6.114 

10 

0.6114 

Error  (Taskload) 

2.341 

10 

0.2341 

Error  (Control  Method  *  Event  Type) 

6.205 

10 

0.6205 

Error  (Control  Method  *  Taskload) 

7.341 

10 

0.7341 

Error  (Event  Type  *  Taskload) 

4.614 

10 

0.4614 

Error  (Control  Method  *  Event  Type  *  Taskload) 

2.068 

10 

0.2068 

Table  26.  ANOVA:  Frustration 


Source 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

Control  Method 

2.909 

1 

2.9090 

4.103 

0.070 

Event  Type 

1.636 

1 

1.6360 

2.222 

0.167 

Taskload 

2.909 

1 

2.9090 

7.111 

0.024 

Control  Method  *  Event  Type 

0.727 

1 

0.7270 

0.434 

0.525 

Control  Method  *  Taskload 

0.000 

1 

0.0000 

0.000 

1.000 

Event  Type  *  Taskload 

0.000 

1 

0.0000 

0.000 

1.000 

Control  Method  *  Event  Type  *  Taskload 

0.182 

1 

0.1820 

0.645 

0.441 

Error  (Control  Method) 

7.091 

10 

0.7091 

Error  (Event  Type) 

7.364 

10 

0.7364 

Error  (Taskload) 

4.091 

10 

0.4091 

Error  (Control  Method  *  Event  Type) 

16.773 

10 

1.6773 

Error  (Control  Method  *  Taskload) 

2.500 

10 

0.2500 

Error  (Event  Type  *  Taskload) 

3.500 

10 

0.3500 

Error  (Control  Method  *  Event  Type  *  Taskload) 

2.818 

10 

0.2818 
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